Oct 31 12:24:10.740173 ip-10-0-140-137 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 12:24:10.740186 ip-10-0-140-137 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 12:24:10.740195 ip-10-0-140-137 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 12:24:10.740533 ip-10-0-140-137 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 12:24:20.948444 ip-10-0-140-137 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 12:24:20.948465 ip-10-0-140-137 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e56e40ab36404186b1edabcb38693e80 --
Oct 31 12:26:37.942922 ip-10-0-140-137 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 12:26:38.390982 ip-10-0-140-137 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:26:38.390982 ip-10-0-140-137 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 12:26:38.390982 ip-10-0-140-137 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:26:38.390982 ip-10-0-140-137 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:26:38.398695 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398596    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:38.398695 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398690    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:38.398695 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398694    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:38.398695 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398698    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:38.398695 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398701    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398704    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398707    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398709    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398712    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398715    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398718    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398721    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398723    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398726    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398729    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398731    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398734    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398736    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398739    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398741    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398744    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398747    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398749    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398752    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:38.398837 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398754    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398759    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398770    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398774    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398777    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398781    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398784    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398786    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398789    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398792    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398794    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398797    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398800    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398803    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398806    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398809    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398811    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398814    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398817    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:38.399306 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398820    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398822    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398825    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398828    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398830    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398833    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398835    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398839    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398842    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398844    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398847    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398850    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398852    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398855    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398858    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398860    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398865    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398867    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398870    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:38.399830 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398873    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398875    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398878    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398880    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398883    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398886    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398888    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398891    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398895    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398898    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398901    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398904    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398906    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398909    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398911    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398914    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398917    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398919    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398922    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:38.400273 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398924    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398927    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398932    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398936    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398939    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398942    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398945    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398948    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398950    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398953    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398955    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398958    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398961    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398963    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398966    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398969    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398971    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398975    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398978    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398980    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:38.400750 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398983    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398986    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398989    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398992    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.398994    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399807    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399815    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399819    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399821    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399824    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399827    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399829    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399833    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399836    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399838    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399841    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399844    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399847    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399850    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399852    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:38.401252 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399855    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399857    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399860    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399863    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399865    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399868    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399870    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399873    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399875    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399878    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399880    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399883    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399886    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399890    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399893    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399896    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399898    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399901    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399903    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:38.401745 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399906    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399909    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399911    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399914    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399917    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399920    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399922    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399925    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399927    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399930    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399933    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399935    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399938    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399941    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399943    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399946    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399948    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399951    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399953    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399956    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:38.402194 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399958    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399961    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399963    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399966    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399968    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399971    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399975    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399977    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399980    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399983    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399985    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399988    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399990    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399993    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399995    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.399998    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400000    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400003    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400006    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400008    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:38.402689 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400011    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400013    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400016    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400019    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400021    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400024    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400027    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400029    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400031    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400034    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400036    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400039    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400041    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400044    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400046    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400050    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400054    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400057    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400061    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400064    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:38.403167 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400066    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400069    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400073    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400077    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400080    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400083    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400086    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400089    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400091    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400094    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400096    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400099    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400614    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400620    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400624    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400626    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400643    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400646    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400649    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:38.403652 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400652    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400654    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400657    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400660    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400663    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400665    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400668    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400671    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400673    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400676    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400679    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400683    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400687    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400690    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400693    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400696    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400699    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400702    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400704    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:38.404103 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400707    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400710    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400712    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400715    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400718    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400720    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400723    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400725    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400728    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400738    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400741    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400744    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400747    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400749    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400752    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400755    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400757    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400760    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400763    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400765    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:38.404563 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400768    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400771    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400773    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400776    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400780    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400782    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400785    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400787    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400790    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400792    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400795    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400798    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400801    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400803    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400806    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400808    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400811    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400813    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400815    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400819    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:38.405079 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400822    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400824    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400834    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400838    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400841    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400844    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400847    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400850    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400853    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400855    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400858    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400861    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400864    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400866    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400869    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400872    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400877    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400880    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400882    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:38.405669 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400885    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400888    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400890    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400893    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400895    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400898    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400901    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400903    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400906    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400909    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400911    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400913    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400916    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400918    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400921    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400924    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400932    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400935    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400938    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:38.406124 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400942    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.400944    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402478    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402489    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402501    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402506    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402511    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402514    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402518    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402523    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402528    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402531    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402534    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402538    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402541    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402545    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402548    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402551    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402554    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402557    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402564    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402567    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402570    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402573    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 12:26:38.406590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402577    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402584    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402587    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402590    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402594    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402597    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402607    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402610    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402614    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402617    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402621    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402625    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402628    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402640    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402643    2577 flags.go:64] FLAG: --enable-server="true"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402647    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402655    2577 flags.go:64] FLAG: --event-burst="100"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402658    2577 flags.go:64] FLAG: --event-qps="50"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402663    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402666    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402672    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402676    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402679    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402682    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402685    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 31 12:26:38.407200 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402688    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402691    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402694    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402698    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402701    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402704    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402707    2577 flags.go:64] FLAG: --feature-gates=""
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402711    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402715    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402718    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402721    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402724    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402727    2577 flags.go:64] FLAG: --help="false"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402730    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402739    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402743    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402746    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402749    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402753    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402756    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402759    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402762    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402765    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 12:26:38.407870 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402768    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402771    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402775    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402778    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402781    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402785    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402787    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402790    2577 flags.go:64] FLAG: --lock-file=""
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402793    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402796    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402799    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402804    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402808    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402811    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402813    2577 flags.go:64] FLAG: --logging-format="text"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402816    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402820    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402823    2577 flags.go:64] FLAG: --manifest-url=""
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402826    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402831    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402834    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402838    2577 flags.go:64] FLAG: --max-pods="110"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402841    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402844    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402854    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 12:26:38.408422 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402857    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402860    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402863    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402866    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402873    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402876    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402879    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402882    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402885    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402889    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402892    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402895    2577 flags.go:64] FLAG: --port="10250"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402898    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402901    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0504251c858d931d3"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402904    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402907    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402910    2577 flags.go:64] FLAG: --register-node="true"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402913    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402917    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402920    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402923    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402926    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402936    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402939    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402943    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402945    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 12:26:38.409055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402948    2577 flags.go:64] FLAG: --runonce="false"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402951    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402954    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402957    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402960    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402963    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402972    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402975    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402979    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402982    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402985    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402988    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402991    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402994    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.402997    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403000    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403005    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403008    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403011    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403017    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403020    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403023    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403026    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403029    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403032    2577 flags.go:64] FLAG: --v="2"
Oct 31 12:26:38.409767 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403036    2577 flags.go:64] FLAG: --version="false"
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403041    2577 flags.go:64] FLAG: --vmodule=""
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403045    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403048    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403246    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403249    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403252    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403255    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403257    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403260    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403263    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403266    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403268    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403271    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403279    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403282    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403285    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403287    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403290    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403292    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:38.410464 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403295    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403298    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403300    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403303    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403306    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403309    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403311    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403314    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403316    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403319    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403322    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403324    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403327    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403329    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403332    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403335    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403337    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403340    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403342    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:38.411015 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403345    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403347    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403350    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403352    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403355    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403357    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403360    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403362    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403371    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403373    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403376    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403378    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403381    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403384    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403387    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403394    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403397    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403402    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403404    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403407    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:38.411503 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403409    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403412    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403415    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403421    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403423    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403426    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403429    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403432    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403434    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403437    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403440    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403442    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403445    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403447    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403450    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403453    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403455    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403458    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403461    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:38.412104 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403463    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403466    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403474    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403477    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403480    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403483    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403487    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403491    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403495    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403498    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403501    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403504    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403507    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403509    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403512    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403515    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403518    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403520    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403523    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:38.412619 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403527    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403531    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403534    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403537    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403540    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403543    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403546    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403548    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403551    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403553    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403556    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403559    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403561    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.403587    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403643    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403648    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:38.413170 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403657    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403660    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403664    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403666    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403669    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403673    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403676    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403678    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403682    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403685    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403688    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403690    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403693    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403696    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403698    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403701    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403704    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403706    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403709    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:38.413571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403712    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403714    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403717    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403719    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403722    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403724    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403727    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403730    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403732    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403735    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403738    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403740    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403743    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403746    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403755    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403758    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403761    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403764    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403768    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403771    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:38.414047 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403773    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403777    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403780    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403782    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403785    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403787    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403790    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403793    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403795    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403798    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403801    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403804    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403806    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403809    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403812    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403814    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403817    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403820    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403822    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403825    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:38.414554 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403827    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403830    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403833    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403836    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403838    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403841    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403844    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403852    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403855    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403859    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403863    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403867    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403870    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403875    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403878    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403881    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403884    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403886    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403889    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:38.415142 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403892    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403895    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403898    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403900    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403903    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403907    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403912    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403915    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403918    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403920    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403923    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403926    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403929    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403932    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403934    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403937    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403939    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403942    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403944    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:38.415966 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403947    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403950    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403962    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403965    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403969    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403972    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.403980    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.411075    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.411090    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.411104    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.411109    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411176    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411182    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411185    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411188    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411190    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411193    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411196    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411199    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411202    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:38.416675 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411204    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411207    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411210    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411212    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411215    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411217    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411220    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411223    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411225    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411228    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411231    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411234    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411236    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411240    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411246    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411249    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411252    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411255    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411258    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411261    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:38.417204 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411264    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411267    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411271    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411275    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411278    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411281    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411284    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411286    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411289    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411292    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411295    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411297    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411300    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411303    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411306    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411308    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411311    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411313    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411316    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:38.417717 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411319    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411322    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411325    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411327    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411330    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411332    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411335    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411338    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411341    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411344    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411347    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411350    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411353    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411356    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411358    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411361    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411364    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411367    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411369    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411372    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:38.418184 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411375    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411377    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411380    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411382    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411385    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411388    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411391    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411393    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411396    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411398    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411401    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411404    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411406    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411409    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411412    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411414    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411417    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411420    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411422    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:38.418834 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411425    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411427    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411436    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411439    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411442    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411446    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411449    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411452    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411455    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411457    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411460    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411463    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411465    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411468    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411472    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411474    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411477    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411479    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:38.419283 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411482    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.411510    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411676    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411682    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411686    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411689    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411692    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411695    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411698    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411701    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411704    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411706    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411709    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411711    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411714    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411716    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:38.419736 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411719    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411722    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411725    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411728    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411731    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411734    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411736    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411739    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411742    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411744    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411747    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411750    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411753    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411756    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411759    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411761    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411764    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411767    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411770    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411772    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:38.420131 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411775    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411777    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411780    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411782    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411785    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411788    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411791    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411794    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411796    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411799    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411802    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411804    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411807    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411810    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411813    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411816    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411819    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411822    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411824    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411827    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:38.420593 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411830    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411833    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411835    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411838    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411841    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411843    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411846    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411849    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411851    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411854    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411856    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411859    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411862    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411864    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411867    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411869    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411874    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411877    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411881    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:38.421089 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411884    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411887    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411890    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411892    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411896    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411900    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411903    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411906    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411909    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411912    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411915    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411917    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411920    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411922    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411925    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411927    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411930    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411932    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411935    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:38.421552 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411937    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411940    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411943    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411945    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411948    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411950    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411953    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411955    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411958    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411960    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411963    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411965    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411968    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.411971    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.411995    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412028    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:26:38.422011 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412032    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412036    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412038    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412041    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412044    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412047    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412050    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412053    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412055    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412058    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412061    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412064    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412066    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412069    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412071    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412074    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412077    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412080    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412082    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412085    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:26:38.422410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412087    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412090    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412092    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412095    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412098    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412101    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412103    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412106    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412109    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412112    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412114    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412117    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412120    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412122    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412125    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412128    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412130    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412133    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412136    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412139    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:26:38.422998 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412141    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412144    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412146    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412149    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412152    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412155    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412158    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412160    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412163    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412165    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412168    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412171    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412173    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412175    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412178    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412180    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412183    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412186    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412189    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412191    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:26:38.423473 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412194    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412197    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412199    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412202    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412206    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412208    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412211    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412214    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412216    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412219    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412222    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412228    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412231    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412234    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412237    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412239    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412242    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412244    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412247    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412249    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:26:38.423952 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412252    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412254    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412257    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412259    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412262    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412264    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412267    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412271    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412275    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412278    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412281    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412284    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412289    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412292    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412296    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412299    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412303    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412306    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:26:38.424410 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412309    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412312    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412314    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412317    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412320    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412322    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:38.412325    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.412378    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.416933    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.417945    2577 server.go:1014] "Starting client certificate rotation"
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.418039    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.418067    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.420164    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 12:26:38.424845 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.421322    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 12:26:38.433064 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.433038    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 31 12:26:38.437789 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.437774    2577 log.go:25] "Validated CRI v1 image API"
Oct 31 12:26:38.438703 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.438654    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 12:26:38.442459 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.442435    2577 fs.go:135] Filesystem UUIDs: map[4e5a7c99-e7fa-4f42-8971-e54d56bd69d6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 83d322fb-9b6a-4354-ae1d-eb03037c0058:/dev/nvme0n1p3]
Oct 31 12:26:38.442545 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.442458    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 12:26:38.448277 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.448161    2577 manager.go:217] Machine: {Timestamp:2026-10-31 12:26:38.446682804 +0000 UTC m=+0.375670303 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116102 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f70d8964258dbd79cc017389f7f9 SystemUUID:ec24f70d-8964-258d-bd79-cc017389f7f9 BootID:e56e40ab-3640-4186-b1ed-abcb38693e80 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d9:68:d1:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d9:68:d1:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:dc:4e:8e:30:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 12:26:38.448921 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.448908    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 12:26:38.449043 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.449018    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 12:26:38.452521 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.452496    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 12:26:38.452690 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.452529    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-137.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 12:26:38.452741 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.452699    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 12:26:38.452741 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.452707    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 12:26:38.452741 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.452718    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 12:26:38.452741 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.452730    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 12:26:38.453788 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.453769    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 12:26:38.453890 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.453881    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 12:26:38.457209 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.457198    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 12:26:38.457330 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.457321    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 12:26:38.457368 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.457360    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 12:26:38.457396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.457373    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 12:26:38.457396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.457383    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 12:26:38.457396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.457395    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 31 12:26:38.457488 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.457403    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 12:26:38.458102 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.458088    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 12:26:38.462493 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.462148    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 12:26:38.469356 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.469341    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 12:26:38.470231 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.470219    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 12:26:38.470273 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.470249    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 12:26:38.470407 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.470396    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 12:26:38.470441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.470414    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 12:26:38.470441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.470420    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 12:26:38.470441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.470426    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 12:26:38.470441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.470431    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 12:26:38.470441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.470437    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 12:26:38.471208 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.471199    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 12:26:38.471237 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.471209    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 12:26:38.471237 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.471216    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 12:26:38.471237 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.471222    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 12:26:38.471237 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.471231    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 12:26:38.471444 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.471240    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 12:26:38.472224 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.472189    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 12:26:38.472224 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.472199    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 12:26:38.475564 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.475544    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-137.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 12:26:38.476249 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.476229    2577 server.go:1263] "Started kubelet"
Oct 31 12:26:38.476343 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.476279    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 12:26:38.476398 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.476320    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 12:26:38.476451 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.476408    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 12:26:38.477051 ip-10-0-140-137 systemd[1]: Started Kubernetes Kubelet.
Oct 31 12:26:38.478048 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.477881    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 12:26:38.478514 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.478500    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 12:26:38.482274 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.482256    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 12:26:38.485678 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.485662    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-54qwb"
Oct 31 12:26:38.486417 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.486402    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 12:26:38.487050 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.487034    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 12:26:38.487275 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.487127    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:38.487357 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.487336    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-137.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 12:26:38.487835 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.487806    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 12:26:38.487930 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.487859    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 12:26:38.487930 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.487876    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 12:26:38.487930 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.487910    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 12:26:38.487930 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.487917    2577 factory.go:55] Registering systemd factory
Oct 31 12:26:38.487930 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.487933    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 31 12:26:38.488105 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.487959    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 12:26:38.488105 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.487968    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 12:26:38.488162 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.488137    2577 factory.go:153] Registering CRI-O factory
Oct 31 12:26:38.488162 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.488152    2577 factory.go:223] Registration of the crio container factory successfully
Oct 31 12:26:38.488233 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.488219    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 12:26:38.488321 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.488306    2577 factory.go:103] Registering Raw factory
Oct 31 12:26:38.488321 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.488324    2577 manager.go:1199] Started watching for new ooms in manager
Oct 31 12:26:38.488778 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.487515    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-137.us-east-2.compute.internal.18e39ce85fbdb6c5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-137.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-137.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:26:38.476203717 +0000 UTC m=+0.405191215,LastTimestamp:2026-10-31 12:26:38.476203717 +0000 UTC m=+0.405191215,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-137.us-east-2.compute.internal,}"
Oct 31 12:26:38.489426 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.489398    2577 manager.go:319] Starting recovery of all containers
Oct 31 12:26:38.499930 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.499889    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-54qwb"
Oct 31 12:26:38.500114 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.500083    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-137.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 12:26:38.501935 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.501913    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 12:26:38.503152 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.503135    2577 manager.go:324] Recovery completed
Oct 31 12:26:38.504621 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.504602    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 31 12:26:38.507519 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.507503    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:38.509374 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.509355    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:38.509454 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.509389    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:38.509454 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.509404    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:38.510006 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.509993    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 31 12:26:38.510006 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.510004    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 12:26:38.510115 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.510017    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:38.510115 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.510019    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 12:26:38.510674 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.510655    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:38.510758 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.510689    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:38.510758 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.510704    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:38.511028 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.511016    2577 policy_none.go:50] "Start"
Oct 31 12:26:38.511093 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.511031    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 12:26:38.511093 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.511045    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 12:26:38.511865 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.511853    2577 policy_none.go:44] "Start"
Oct 31 12:26:38.538423 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.538379    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 12:26:38.541107 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.541093    2577 manager.go:341] "Starting Device Plugin manager"
Oct 31 12:26:38.541799 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.541779    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 12:26:38.541874 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.541802    2577 server.go:87] "Starting device plugin registration server"
Oct 31 12:26:38.542150 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.542137    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 12:26:38.542201 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.542153    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 12:26:38.542310 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.542257    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 12:26:38.542350 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.542332    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 12:26:38.542350 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.542344    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 12:26:38.542897 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.542867    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 12:26:38.543008 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.542927    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:38.622685 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.622656    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 12:26:38.623909 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.623887    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 12:26:38.624307 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.624292    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 12:26:38.624392 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.624325    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 12:26:38.624392 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.624374    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 12:26:38.627786 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.627770    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:26:38.642858 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.642810    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:38.644733 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.644720    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:38.644799 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.644750    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:38.644799 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.644760    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:38.644799 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.644784    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.653043 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.653029    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.653093 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.653052    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-137.us-east-2.compute.internal\": node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:38.690563 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.690395    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:38.725174 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.725143    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"]
Oct 31 12:26:38.725309 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.725208    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:38.726137 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.726119    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:38.726233 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.726152    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:38.726233 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.726162    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:38.727301 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.727286    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:38.727498 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.727483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.727566 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.727537    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:38.728051 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.728038    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:38.728135 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.728065    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:38.728135 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.728081    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:38.728281 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.728267    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:38.728324 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.728297    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:38.728324 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.728308    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:38.729138 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.729123    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.729206 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.729170    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:26:38.729789 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.729774    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:26:38.729874 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.729801    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:26:38.729874 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.729815    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:26:38.754913 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.754888    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found" node="ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.759135 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.759117    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found" node="ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.789662 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.789617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5b201abd7a627e01510f6952236144-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"4f5b201abd7a627e01510f6952236144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.789806 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.789666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d0c123eab79dd4d273b2febcaaf1af3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"0d0c123eab79dd4d273b2febcaaf1af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.789806 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.789695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f5b201abd7a627e01510f6952236144-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"4f5b201abd7a627e01510f6952236144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.791443 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.791426    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:38.890628 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.890592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f5b201abd7a627e01510f6952236144-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"4f5b201abd7a627e01510f6952236144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.890819 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.890658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5b201abd7a627e01510f6952236144-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"4f5b201abd7a627e01510f6952236144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.890819 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.890675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d0c123eab79dd4d273b2febcaaf1af3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"0d0c123eab79dd4d273b2febcaaf1af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.890819 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.890677    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f5b201abd7a627e01510f6952236144-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"4f5b201abd7a627e01510f6952236144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.890819 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.890724    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d0c123eab79dd4d273b2febcaaf1af3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"0d0c123eab79dd4d273b2febcaaf1af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.890819 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:38.890736    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5b201abd7a627e01510f6952236144-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"4f5b201abd7a627e01510f6952236144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:38.891646 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.891618    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:38.992381 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:38.992323    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:39.057468 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:39.057446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:39.061967 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:39.061950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:39.093229 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:39.093206    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:40.819497 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:40.819462    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:40.919937 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:40.919867    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:41.020325 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:41.020287    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:41.043441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:41.043419    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 12:26:41.043561 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:41.043549    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 12:26:41.120539 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:41.120506    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:41.134612 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:41.134577    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:21:40 +0000 UTC" deadline="2028-06-12 16:21:11.43025117 +0000 UTC"
Oct 31 12:26:41.134612 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:41.134604    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14163h54m30.295650237s"
Oct 31 12:26:41.221108 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:41.221077    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:41.321474 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:41.321443    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-137.us-east-2.compute.internal\" not found"
Oct 31 12:26:41.321609 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:41.321512    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:26:41.324509 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:41.324484    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5b201abd7a627e01510f6952236144.slice/crio-08191ab633083238796092e5c3633512d3b6eb190d2f27ed6431ff7014628ede WatchSource:0}: Error finding container 08191ab633083238796092e5c3633512d3b6eb190d2f27ed6431ff7014628ede: Status 404 returned error can't find the container with id 08191ab633083238796092e5c3633512d3b6eb190d2f27ed6431ff7014628ede
Oct 31 12:26:41.329831 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:41.329812    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:26:41.335416 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:41.335396    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0c123eab79dd4d273b2febcaaf1af3.slice/crio-b957d88582d3c25c206d8852f928e9f9ad8952db75bc492b45f7dc21f76aaf68 WatchSource:0}: Error finding container b957d88582d3c25c206d8852f928e9f9ad8952db75bc492b45f7dc21f76aaf68: Status 404 returned error can't find the container with id b957d88582d3c25c206d8852f928e9f9ad8952db75bc492b45f7dc21f76aaf68
Oct 31 12:26:41.346182 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:41.345555    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:26:41.413054 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:41.413015    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:41.424283 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:41.424252    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 12:26:41.425299 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:41.425284    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:26:41.435857 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:41.435839    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 12:26:41.681664 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:41.681589    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:26:42.084075 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.084002    2577 apiserver.go:52] "Watching apiserver"
Oct 31 12:26:42.091482 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.091454    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 12:26:42.091831 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.091813    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-s9njc","openshift-multus/multus-additional-cni-plugins-lxnjj","openshift-network-diagnostics/network-check-target-jhwnt","kube-system/global-pull-secret-syncer-p8vf7","kube-system/konnectivity-agent-blk6p","openshift-cluster-node-tuning-operator/tuned-5xw6s","openshift-dns/node-resolver-bw8kq","openshift-ovn-kubernetes/ovnkube-node-lnkms","kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal","openshift-multus/multus-2wtl6","openshift-multus/network-metrics-daemon-vqlp8","openshift-network-operator/iptables-alerter-z42qt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"]
Oct 31 12:26:42.094305 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.094276    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:26:42.096582 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.096564    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:26:42.099488 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.099467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:42.099614 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.099548    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:26:42.099685 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.099668    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9d2j7\""
Oct 31 12:26:42.099853 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.099837    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 12:26:42.099902 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.099868    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 12:26:42.099902 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.099880    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 12:26:42.099902 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.099843    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 12:26:42.099902 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.099885    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:26:42.100042 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.099914    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ng46x\""
Oct 31 12:26:42.100995 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.100974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:42.101116 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.101045    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:26:42.103272 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.103252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:26:42.105610 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.105591    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.107828 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.107810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.113154 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.113136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:26:42.113249 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.113224    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 12:26:42.115574 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.115557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.117910 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.117883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.120156 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.120138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.123926 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.123847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:42.124042 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.123923    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:26:42.125874 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.125852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9d44df-91e2-4e21-afd7-eac98bf106e5-tmp-dir\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:26:42.125974 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.125887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/119a3a49-2f9f-4285-bba5-e035d61b03cd-host-slash\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:26:42.125974 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.125911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysconfig\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.125974 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.125938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe072ecb-08a7-4c60-ad8b-d6643882ba73-serviceca\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:26:42.126091 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-system-cni-dir\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.126091 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.126091 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.126202 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.126202 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.126202 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l822j\" (UniqueName: \"kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j\") pod \"network-check-target-jhwnt\" (UID: \"23cc7cc6-1673-434b-a56f-e83298dc5d91\") " pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:42.126323 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-kubernetes\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.126323 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4958\" (UniqueName: \"kubernetes.io/projected/fe072ecb-08a7-4c60-ad8b-d6643882ba73-kube-api-access-m4958\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:26:42.126416 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-device-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.126416 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.126482 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxv7f\" (UniqueName: \"kubernetes.io/projected/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-kube-api-access-cxv7f\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.126482 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gljhb\" (UniqueName: \"kubernetes.io/projected/119a3a49-2f9f-4285-bba5-e035d61b03cd-kube-api-access-gljhb\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:26:42.126546 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-modprobe-d\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.126546 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysctl-d\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.126609 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysctl-conf\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.126609 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-systemd\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.126676 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-run\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.126711 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-host\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.126711 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cnibin\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.126786 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cni-binary-copy\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.126786 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-kubelet-config\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:42.126924 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-dbus\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:42.126924 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:42.126924 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126846    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab9d44df-91e2-4e21-afd7-eac98bf106e5-hosts-file\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:26:42.127023 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/119a3a49-2f9f-4285-bba5-e035d61b03cd-iptables-alerter-script\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:26:42.127023 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jk2d\" (UniqueName: \"kubernetes.io/projected/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-kube-api-access-4jk2d\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.127023 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8fc463-aafe-4750-aaeb-827522735a5e-konnectivity-ca\") pod \"konnectivity-agent-blk6p\" (UID: \"0b8fc463-aafe-4750-aaeb-827522735a5e\") " pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:26:42.127023 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.126984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-registration-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.127023 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.127016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-lib-modules\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.127209 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.127033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-var-lib-kubelet\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.127209 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.127058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-httpx\" (UniqueName: \"kubernetes.io/projected/ab9d44df-91e2-4e21-afd7-eac98bf106e5-kube-api-access-httpx\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:26:42.127209 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.127088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-socket-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.127209 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.127112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-sys-fs\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.127209 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.127132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-sys\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.127209 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.127164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-tuned\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.127209 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.127197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-tmp\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.127493 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.127224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcjhz\" (UniqueName: \"kubernetes.io/projected/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-kube-api-access-kcjhz\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.127493 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.127244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe072ecb-08a7-4c60-ad8b-d6643882ba73-host\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:26:42.127493 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.127283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8fc463-aafe-4750-aaeb-827522735a5e-agent-certs\") pod \"konnectivity-agent-blk6p\" (UID: \"0b8fc463-aafe-4750-aaeb-827522735a5e\") " pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:26:42.150401 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.150383    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 12:26:42.150475 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.150434    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r4lh8"
Oct 31 12:26:42.151899 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.151881    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:26:42.161849 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.161826    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r4lh8"
Oct 31 12:26:42.197919 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.197898    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 12:26:42.198059 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.198047    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 12:26:42.198192 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.198175    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 12:26:42.198269 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.198249    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 12:26:42.198396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.198345    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 12:26:42.198469 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.198455    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m5kv7\""
Oct 31 12:26:42.198848 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.198827    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:26:42.198941 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.198845    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 12:26:42.198941 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.198874    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bbb6f\""
Oct 31 12:26:42.198941 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.198911    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 12:26:42.199138 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199118    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 12:26:42.199138 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199134    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hzb2l\""
Oct 31 12:26:42.199251 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199134    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 12:26:42.199251 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199129    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 12:26:42.199251 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199137    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 12:26:42.199251 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199175    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 12:26:42.199251 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199207    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 12:26:42.199251 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199239    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 12:26:42.199251 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199248    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 12:26:42.199684 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199670    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 12:26:42.199753 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199693    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 12:26:42.199862 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199849    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-df646\""
Oct 31 12:26:42.199952 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.199936    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 12:26:42.200237 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.200216    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 12:26:42.200313 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.200253    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 12:26:42.200553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.200518    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kldqp\""
Oct 31 12:26:42.200663 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.200543    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8ljk6\""
Oct 31 12:26:42.200663 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.200591    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 12:26:42.200663 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.200596    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x5rvd\""
Oct 31 12:26:42.215054 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.215030    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 12:26:42.227470 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httpx\" (UniqueName: \"kubernetes.io/projected/ab9d44df-91e2-4e21-afd7-eac98bf106e5-kube-api-access-httpx\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:26:42.227594 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-socket-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.227594 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-sys-fs\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.227594 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-sys\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.227594 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-tuned\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.227798 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-tmp\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.227798 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227622    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-sys\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.227798 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe072ecb-08a7-4c60-ad8b-d6643882ba73-host\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:26:42.227798 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-cni-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.227798 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227669    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe072ecb-08a7-4c60-ad8b-d6643882ba73-host\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:26:42.227798 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-kubelet\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.227798 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-etc-kubernetes\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.227798 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-ovn\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.227798 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.227798 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/119a3a49-2f9f-4285-bba5-e035d61b03cd-host-slash\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:26:42.227798 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-systemd-units\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227903    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-sys-fs\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227892    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovn-node-metrics-cert\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.227969    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/119a3a49-2f9f-4285-bba5-e035d61b03cd-host-slash\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228098    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-socket-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4n6f\" (UniqueName: \"kubernetes.io/projected/7287c3ee-7c75-47e2-9c34-d436193f32e3-kube-api-access-c4n6f\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-kubernetes\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-netns\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gljhb\" (UniqueName: \"kubernetes.io/projected/119a3a49-2f9f-4285-bba5-e035d61b03cd-kube-api-access-gljhb\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:26:42.228396 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysctl-d\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysctl-conf\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-kubelet\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-systemd\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-run\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228516    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysctl-d\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-host\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228562    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-kubernetes\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cni-binary-copy\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228582    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysctl-conf\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-kubelet-config\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228607    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228627    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-systemd\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228677    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-kubelet-config\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.228746    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228746    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-host\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.229065 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228755    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-run\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.228863    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret podName:8c53a9b9-23b0-4598-8ee6-35de80dd12d4 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:42.728842487 +0000 UTC m=+3.032110135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret") pod "global-pull-secret-syncer-p8vf7" (UID: "8c53a9b9-23b0-4598-8ee6-35de80dd12d4") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxxz\" (UniqueName: \"kubernetes.io/projected/95feab9f-adf7-4d34-bde6-d0a1b450df6c-kube-api-access-kvxxz\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-systemd\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab9d44df-91e2-4e21-afd7-eac98bf106e5-hosts-file\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.228989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/119a3a49-2f9f-4285-bba5-e035d61b03cd-iptables-alerter-script\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jk2d\" (UniqueName: \"kubernetes.io/projected/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-kube-api-access-4jk2d\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-cni-bin\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-run-netns\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-var-lib-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovnkube-config\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-env-overrides\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-lib-modules\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229236    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-cni-multus\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229293    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab9d44df-91e2-4e21-afd7-eac98bf106e5-hosts-file\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:26:42.229893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-etc-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8fc463-aafe-4750-aaeb-827522735a5e-agent-certs\") pod \"konnectivity-agent-blk6p\" (UID: \"0b8fc463-aafe-4750-aaeb-827522735a5e\") " pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-k8s-cni-cncf-io\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-multus-certs\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcjhz\" (UniqueName: \"kubernetes.io/projected/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-kube-api-access-kcjhz\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-system-cni-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229578    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/119a3a49-2f9f-4285-bba5-e035d61b03cd-iptables-alerter-script\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229605    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-lib-modules\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229643    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cni-binary-copy\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-socket-dir-parent\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-hostroot\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9d44df-91e2-4e21-afd7-eac98bf106e5-tmp-dir\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysconfig\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe072ecb-08a7-4c60-ad8b-d6643882ba73-serviceca\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.229910    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230001    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysconfig\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-system-cni-dir\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.230672 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230129    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-system-cni-dir\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-slash\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l822j\" (UniqueName: \"kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j\") pod \"network-check-target-jhwnt\" (UID: \"23cc7cc6-1673-434b-a56f-e83298dc5d91\") " pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4958\" (UniqueName: \"kubernetes.io/projected/fe072ecb-08a7-4c60-ad8b-d6643882ba73-kube-api-access-m4958\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8gb9\" (UniqueName: \"kubernetes.io/projected/5a733526-fcbe-483b-8e2a-e58c99be5707-kube-api-access-h8gb9\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-device-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxv7f\" (UniqueName: \"kubernetes.io/projected/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-kube-api-access-cxv7f\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-modprobe-d\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-daemon-config\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-cni-bin\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cnibin\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230460    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9d44df-91e2-4e21-afd7-eac98bf106e5-tmp-dir\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230456    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe072ecb-08a7-4c60-ad8b-d6643882ba73-serviceca\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230517    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-dbus\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:42.231429 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8fc463-aafe-4750-aaeb-827522735a5e-konnectivity-ca\") pod \"konnectivity-agent-blk6p\" (UID: \"0b8fc463-aafe-4750-aaeb-827522735a5e\") " pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:26:42.232215 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovnkube-script-lib\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.232215 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230662    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.232215 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230754    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-device-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.232215 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230835    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cnibin\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.232215 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.230857    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-dbus\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:42.232215 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.231044    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-modprobe-d\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.232215 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.231466    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8fc463-aafe-4750-aaeb-827522735a5e-konnectivity-ca\") pod \"konnectivity-agent-blk6p\" (UID: \"0b8fc463-aafe-4750-aaeb-827522735a5e\") " pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:26:42.232691 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.232670    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-tmp\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.232751 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.232707    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-tuned\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.232904 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.232881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a733526-fcbe-483b-8e2a-e58c99be5707-cni-binary-copy\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.232985 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.232933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-conf-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.232985 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.232958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-node-log\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.233126 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.232983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-log-socket\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.233126 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.233013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-registration-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.233126 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.233038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-var-lib-kubelet\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.233126 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.233066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-cnibin\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.233126 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.233081    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-registration-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.233126 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.233091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-cni-netd\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.233434 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.233147    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-var-lib-kubelet\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.235076 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.234657    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8fc463-aafe-4750-aaeb-827522735a5e-agent-certs\") pod \"konnectivity-agent-blk6p\" (UID: \"0b8fc463-aafe-4750-aaeb-827522735a5e\") " pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:26:42.238773 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.238751    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-httpx\" (UniqueName: \"kubernetes.io/projected/ab9d44df-91e2-4e21-afd7-eac98bf106e5-kube-api-access-httpx\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:26:42.241841 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.241807    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:26:42.241841 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.241830    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:26:42.241841 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.241844    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l822j for pod openshift-network-diagnostics/network-check-target-jhwnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:42.242090 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.241909    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j podName:23cc7cc6-1673-434b-a56f-e83298dc5d91 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:42.741890691 +0000 UTC m=+3.045158342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l822j" (UniqueName: "kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j") pod "network-check-target-jhwnt" (UID: "23cc7cc6-1673-434b-a56f-e83298dc5d91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:42.242489 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.242469    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gljhb\" (UniqueName: \"kubernetes.io/projected/119a3a49-2f9f-4285-bba5-e035d61b03cd-kube-api-access-gljhb\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:26:42.243838 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.243815    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcjhz\" (UniqueName: \"kubernetes.io/projected/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-kube-api-access-kcjhz\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.245432 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.245412    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxv7f\" (UniqueName: \"kubernetes.io/projected/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-kube-api-access-cxv7f\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.245954 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.245777    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jk2d\" (UniqueName: \"kubernetes.io/projected/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-kube-api-access-4jk2d\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.245954 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.245796    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4958\" (UniqueName: \"kubernetes.io/projected/fe072ecb-08a7-4c60-ad8b-d6643882ba73-kube-api-access-m4958\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:26:42.256923 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.256873    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal" event={"ID":"0d0c123eab79dd4d273b2febcaaf1af3","Type":"ContainerStarted","Data":"b957d88582d3c25c206d8852f928e9f9ad8952db75bc492b45f7dc21f76aaf68"}
Oct 31 12:26:42.257957 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.257935    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal" event={"ID":"4f5b201abd7a627e01510f6952236144","Type":"ContainerStarted","Data":"08191ab633083238796092e5c3633512d3b6eb190d2f27ed6431ff7014628ede"}
Oct 31 12:26:42.334172 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-k8s-cni-cncf-io\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334172 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-multus-certs\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334172 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-system-cni-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334172 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-socket-dir-parent\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-hostroot\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-slash\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334223    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-multus-certs\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8gb9\" (UniqueName: \"kubernetes.io/projected/5a733526-fcbe-483b-8e2a-e58c99be5707-kube-api-access-h8gb9\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-daemon-config\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334283    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-k8s-cni-cncf-io\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-cni-bin\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334322    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-hostroot\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovnkube-script-lib\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a733526-fcbe-483b-8e2a-e58c99be5707-cni-binary-copy\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-conf-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334390    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-system-cni-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-node-log\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-log-socket\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334439    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-socket-dir-parent\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-cnibin\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-cni-netd\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-cni-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-kubelet\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-etc-kubernetes\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-ovn\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-systemd-units\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovn-node-metrics-cert\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4n6f\" (UniqueName: \"kubernetes.io/projected/7287c3ee-7c75-47e2-9c34-d436193f32e3-kube-api-access-c4n6f\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-netns\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-kubelet\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxxz\" (UniqueName: \"kubernetes.io/projected/95feab9f-adf7-4d34-bde6-d0a1b450df6c-kube-api-access-kvxxz\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-systemd\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.334866 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-cni-bin\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.335553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-run-netns\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.335553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-var-lib-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.335553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovnkube-config\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.335553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-env-overrides\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.335553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-cni-multus\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.335553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-etc-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.335553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.334893    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-etc-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.335553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.335054    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovnkube-script-lib\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.335553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.335099    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-slash\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.335553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.335283    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a733526-fcbe-483b-8e2a-e58c99be5707-cni-binary-copy\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.335553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.335479    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-netns\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.335553 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.335532    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:42.335914 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.335575    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs podName:95feab9f-adf7-4d34-bde6-d0a1b450df6c nodeName:}" failed. No retries permitted until 2026-10-31 12:26:42.835564604 +0000 UTC m=+3.138832232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs") pod "network-metrics-daemon-vqlp8" (UID: "95feab9f-adf7-4d34-bde6-d0a1b450df6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:42.335914 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.335711    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.335914 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.335741    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-kubelet\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.335914 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.335875    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-systemd\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.335914 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.335902    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-cni-bin\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.336055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.335925    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-run-netns\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.336055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.335949    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-var-lib-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.336055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.335967    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-daemon-config\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.336055 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336023    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-cni-bin\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.336174 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336065    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-cni-multus\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.336300 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336278    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovnkube-config\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.336385 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336370    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-cni-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.336431 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336409    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-env-overrides\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.336463 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336413    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-conf-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.336463 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336453    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-log-socket\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.336526 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336463    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-kubelet\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.336526 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336488    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-cnibin\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.336526 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336497    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-cni-netd\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.336526 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336502    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-node-log\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.336762 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336543    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-etc-kubernetes\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.336762 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336547    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.336762 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336545    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-systemd-units\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.336762 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336575    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.336762 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.336609    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-ovn\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.337638 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.337577    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovn-node-metrics-cert\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.347710 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.347684    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxxz\" (UniqueName: \"kubernetes.io/projected/95feab9f-adf7-4d34-bde6-d0a1b450df6c-kube-api-access-kvxxz\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:42.348413 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.348392    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4n6f\" (UniqueName: \"kubernetes.io/projected/7287c3ee-7c75-47e2-9c34-d436193f32e3-kube-api-access-c4n6f\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.350296 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.350278    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8gb9\" (UniqueName: \"kubernetes.io/projected/5a733526-fcbe-483b-8e2a-e58c99be5707-kube-api-access-h8gb9\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.407952 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.407920    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:26:42.417441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.417415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:26:42.419661 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:42.419624    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9d44df_91e2_4e21_afd7_eac98bf106e5.slice/crio-8f4c70cafe06d740f8605ade053156e1482bb725fcc650779c2bb03b1914ab15 WatchSource:0}: Error finding container 8f4c70cafe06d740f8605ade053156e1482bb725fcc650779c2bb03b1914ab15: Status 404 returned error can't find the container with id 8f4c70cafe06d740f8605ade053156e1482bb725fcc650779c2bb03b1914ab15
Oct 31 12:26:42.424132 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.424110    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:26:42.430046 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.430026    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:26:42.443626 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.443601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:26:42.443821 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.443798    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:26:42.455076 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:42.455026    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be38679_7629_4ec8_aed1_d2ef3fcd8a10.slice/crio-8fd7254459e0d5033b1468aa33d32890f9898ca005969bb0e58c9b8cad878d50 WatchSource:0}: Error finding container 8fd7254459e0d5033b1468aa33d32890f9898ca005969bb0e58c9b8cad878d50: Status 404 returned error can't find the container with id 8fd7254459e0d5033b1468aa33d32890f9898ca005969bb0e58c9b8cad878d50
Oct 31 12:26:42.458281 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:42.458246    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe072ecb_08a7_4c60_ad8b_d6643882ba73.slice/crio-f494df66607706fc97ff0268894b86d3efeac283af0fb58b605eb1ad9583f969 WatchSource:0}: Error finding container f494df66607706fc97ff0268894b86d3efeac283af0fb58b605eb1ad9583f969: Status 404 returned error can't find the container with id f494df66607706fc97ff0268894b86d3efeac283af0fb58b605eb1ad9583f969
Oct 31 12:26:42.461573 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.461548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2wtl6"
Oct 31 12:26:42.470597 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.470573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:26:42.470995 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:42.470979    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a733526_fcbe_483b_8e2a_e58c99be5707.slice/crio-02ae7b8518a5413a0f029f81d85a276fc34e2bb806662d3112a4dc68973b1ea6 WatchSource:0}: Error finding container 02ae7b8518a5413a0f029f81d85a276fc34e2bb806662d3112a4dc68973b1ea6: Status 404 returned error can't find the container with id 02ae7b8518a5413a0f029f81d85a276fc34e2bb806662d3112a4dc68973b1ea6
Oct 31 12:26:42.478517 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.478499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:26:42.478711 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:42.478689    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e1ad7d_c64f_4f2b_b14a_c9b69d97f43a.slice/crio-903f93924da898fa053ab86d157eb948eb84188f5343dbfa4cfeed29fb7211e5 WatchSource:0}: Error finding container 903f93924da898fa053ab86d157eb948eb84188f5343dbfa4cfeed29fb7211e5: Status 404 returned error can't find the container with id 903f93924da898fa053ab86d157eb948eb84188f5343dbfa4cfeed29fb7211e5
Oct 31 12:26:42.487870 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:26:42.487846    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7287c3ee_7c75_47e2_9c34_d436193f32e3.slice/crio-388243f74d05102978315458c3adbd21e98488457e1843d8feceaa5405b877ce WatchSource:0}: Error finding container 388243f74d05102978315458c3adbd21e98488457e1843d8feceaa5405b877ce: Status 404 returned error can't find the container with id 388243f74d05102978315458c3adbd21e98488457e1843d8feceaa5405b877ce
Oct 31 12:26:42.737587 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.737509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:42.737741 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.737672    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:42.737741 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.737734    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret podName:8c53a9b9-23b0-4598-8ee6-35de80dd12d4 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:43.737718513 +0000 UTC m=+4.040986155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret") pod "global-pull-secret-syncer-p8vf7" (UID: "8c53a9b9-23b0-4598-8ee6-35de80dd12d4") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:42.838868 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.838835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:42.839007 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:42.838913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l822j\" (UniqueName: \"kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j\") pod \"network-check-target-jhwnt\" (UID: \"23cc7cc6-1673-434b-a56f-e83298dc5d91\") " pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:42.839056 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.839035    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:26:42.839056 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.839052    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:26:42.839147 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.839065    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l822j for pod openshift-network-diagnostics/network-check-target-jhwnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:42.839147 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.839112    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j podName:23cc7cc6-1673-434b-a56f-e83298dc5d91 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:43.839099113 +0000 UTC m=+4.142366754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l822j" (UniqueName: "kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j") pod "network-check-target-jhwnt" (UID: "23cc7cc6-1673-434b-a56f-e83298dc5d91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:42.839552 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.839530    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:42.839627 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:42.839594    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs podName:95feab9f-adf7-4d34-bde6-d0a1b450df6c nodeName:}" failed. No retries permitted until 2026-10-31 12:26:43.839577775 +0000 UTC m=+4.142845406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs") pod "network-metrics-daemon-vqlp8" (UID: "95feab9f-adf7-4d34-bde6-d0a1b450df6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:43.164734 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.164669    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:21:42 +0000 UTC" deadline="2028-04-13 21:03:19.672563139 +0000 UTC"
Oct 31 12:26:43.164734 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.164730    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12728h36m36.507835827s"
Oct 31 12:26:43.263085 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.263051    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-blk6p" event={"ID":"0b8fc463-aafe-4750-aaeb-827522735a5e","Type":"ContainerStarted","Data":"7ae1e5a6300d7f843f3e921d46b6523eecb9a454926f0d8d30153a72639ebd10"}
Oct 31 12:26:43.265225 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.264874    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z42qt" event={"ID":"119a3a49-2f9f-4285-bba5-e035d61b03cd","Type":"ContainerStarted","Data":"a715c33150a3ce754318231e430b41fff8dc2efee8cd5a01206ad8f2ea204924"}
Oct 31 12:26:43.266544 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.266283    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms" event={"ID":"7287c3ee-7c75-47e2-9c34-d436193f32e3","Type":"ContainerStarted","Data":"388243f74d05102978315458c3adbd21e98488457e1843d8feceaa5405b877ce"}
Oct 31 12:26:43.268683 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.268653    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lxnjj" event={"ID":"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a","Type":"ContainerStarted","Data":"903f93924da898fa053ab86d157eb948eb84188f5343dbfa4cfeed29fb7211e5"}
Oct 31 12:26:43.270848 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.270826    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2wtl6" event={"ID":"5a733526-fcbe-483b-8e2a-e58c99be5707","Type":"ContainerStarted","Data":"02ae7b8518a5413a0f029f81d85a276fc34e2bb806662d3112a4dc68973b1ea6"}
Oct 31 12:26:43.272951 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.272925    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9njc" event={"ID":"fe072ecb-08a7-4c60-ad8b-d6643882ba73","Type":"ContainerStarted","Data":"f494df66607706fc97ff0268894b86d3efeac283af0fb58b605eb1ad9583f969"}
Oct 31 12:26:43.278570 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.277369    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5xw6s" event={"ID":"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9","Type":"ContainerStarted","Data":"27c8f6e5e2982762b2d3651ee5244ca1731f1ef5b81b0affa1c19aeef81efac8"}
Oct 31 12:26:43.293543 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.293467    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bw8kq" event={"ID":"ab9d44df-91e2-4e21-afd7-eac98bf106e5","Type":"ContainerStarted","Data":"8f4c70cafe06d740f8605ade053156e1482bb725fcc650779c2bb03b1914ab15"}
Oct 31 12:26:43.297911 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.297831    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5" event={"ID":"4be38679-7629-4ec8-aed1-d2ef3fcd8a10","Type":"ContainerStarted","Data":"8fd7254459e0d5033b1468aa33d32890f9898ca005969bb0e58c9b8cad878d50"}
Oct 31 12:26:43.748684 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.748643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:43.748847 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:43.748805    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:43.748900 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:43.748871    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret podName:8c53a9b9-23b0-4598-8ee6-35de80dd12d4 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:45.748853438 +0000 UTC m=+6.052121080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret") pod "global-pull-secret-syncer-p8vf7" (UID: "8c53a9b9-23b0-4598-8ee6-35de80dd12d4") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:43.849112 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.849075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:43.849285 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:43.849157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l822j\" (UniqueName: \"kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j\") pod \"network-check-target-jhwnt\" (UID: \"23cc7cc6-1673-434b-a56f-e83298dc5d91\") " pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:43.849369 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:43.849296    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:26:43.849369 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:43.849315    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:26:43.849369 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:43.849329    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l822j for pod openshift-network-diagnostics/network-check-target-jhwnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:43.849520 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:43.849420    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j podName:23cc7cc6-1673-434b-a56f-e83298dc5d91 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:45.849398663 +0000 UTC m=+6.152666326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l822j" (UniqueName: "kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j") pod "network-check-target-jhwnt" (UID: "23cc7cc6-1673-434b-a56f-e83298dc5d91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:43.849520 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:43.849503    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:43.849620 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:43.849540    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs podName:95feab9f-adf7-4d34-bde6-d0a1b450df6c nodeName:}" failed. No retries permitted until 2026-10-31 12:26:45.84952923 +0000 UTC m=+6.152796865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs") pod "network-metrics-daemon-vqlp8" (UID: "95feab9f-adf7-4d34-bde6-d0a1b450df6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:44.165784 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:44.165738    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:21:42 +0000 UTC" deadline="2028-04-26 21:17:49.165702861 +0000 UTC"
Oct 31 12:26:44.165784 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:44.165783    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13040h51m4.999924728s"
Oct 31 12:26:44.254610 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:44.253491    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:44.254610 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:44.253841    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:26:44.254610 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:44.254263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:44.254610 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:44.254372    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:26:44.254610 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:44.254443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:44.254610 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:44.254525    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:26:44.313977 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:44.313033    2577 generic.go:356] "Generic (PLEG): container finished" podID="4f5b201abd7a627e01510f6952236144" containerID="702f596263a146b805d76712e1ff0addb4627fe42cea7ff86bb0b8bf29e4dd2d" exitCode=0
Oct 31 12:26:44.313977 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:44.313092    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal" event={"ID":"4f5b201abd7a627e01510f6952236144","Type":"ContainerDied","Data":"702f596263a146b805d76712e1ff0addb4627fe42cea7ff86bb0b8bf29e4dd2d"}
Oct 31 12:26:45.336805 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:45.336722    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal" event={"ID":"0d0c123eab79dd4d273b2febcaaf1af3","Type":"ContainerStarted","Data":"cf40578246a8f3dbd643a23b5e38641356dd7292a1342f28d4778bc317c5f242"}
Oct 31 12:26:45.344253 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:45.344221    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal" event={"ID":"4f5b201abd7a627e01510f6952236144","Type":"ContainerStarted","Data":"1d591fb833f67be4150ad43177acc28cc476c050c2654d7458f9bf9ce6b90bd8"}
Oct 31 12:26:45.369423 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:45.369347    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal" podStartSLOduration=4.369330107 podStartE2EDuration="4.369330107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:45.352923522 +0000 UTC m=+5.656191175" watchObservedRunningTime="2026-10-31 12:26:45.369330107 +0000 UTC m=+5.672597757"
Oct 31 12:26:45.369574 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:45.369466    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal" podStartSLOduration=4.369458557 podStartE2EDuration="4.369458557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:45.3685768 +0000 UTC m=+5.671844453" watchObservedRunningTime="2026-10-31 12:26:45.369458557 +0000 UTC m=+5.672726214"
Oct 31 12:26:45.766940 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:45.766901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:45.767133 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:45.767103    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:45.767226 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:45.767212    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret podName:8c53a9b9-23b0-4598-8ee6-35de80dd12d4 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:49.767176207 +0000 UTC m=+10.070443852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret") pod "global-pull-secret-syncer-p8vf7" (UID: "8c53a9b9-23b0-4598-8ee6-35de80dd12d4") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:45.867914 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:45.867824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:45.867914 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:45.867910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l822j\" (UniqueName: \"kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j\") pod \"network-check-target-jhwnt\" (UID: \"23cc7cc6-1673-434b-a56f-e83298dc5d91\") " pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:45.868136 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:45.868070    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:26:45.868136 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:45.868096    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:26:45.868136 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:45.868110    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l822j for pod openshift-network-diagnostics/network-check-target-jhwnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:45.868275 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:45.868169    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j podName:23cc7cc6-1673-434b-a56f-e83298dc5d91 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:49.868151304 +0000 UTC m=+10.171418936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l822j" (UniqueName: "kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j") pod "network-check-target-jhwnt" (UID: "23cc7cc6-1673-434b-a56f-e83298dc5d91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:45.868601 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:45.868581    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:45.868681 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:45.868638    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs podName:95feab9f-adf7-4d34-bde6-d0a1b450df6c nodeName:}" failed. No retries permitted until 2026-10-31 12:26:49.868621334 +0000 UTC m=+10.171888965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs") pod "network-metrics-daemon-vqlp8" (UID: "95feab9f-adf7-4d34-bde6-d0a1b450df6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:46.252564 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:46.252494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:46.252715 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:46.252602    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:26:46.252715 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:46.252679    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:46.252807 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:46.252752    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:26:46.252849 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:46.252831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:46.252934 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:46.252915    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:26:46.351752 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:46.351718    2577 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" containerID="7a39e5bb5aa773aeb4f65dfc0ec7a8e22140b61c57d04f0dc18a31b8941223c2" exitCode=0
Oct 31 12:26:46.352706 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:46.352684    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lxnjj" event={"ID":"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a","Type":"ContainerDied","Data":"7a39e5bb5aa773aeb4f65dfc0ec7a8e22140b61c57d04f0dc18a31b8941223c2"}
Oct 31 12:26:47.356996 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:47.356852    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-blk6p" event={"ID":"0b8fc463-aafe-4750-aaeb-827522735a5e","Type":"ContainerStarted","Data":"6ec66854cd9cfa34568d1042cd5e0a1a26925a8578d4c28cbc631f9f0fc134af"}
Oct 31 12:26:47.375135 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:47.374681    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-blk6p" podStartSLOduration=3.465316131 podStartE2EDuration="7.374662835s" totalImagesPullingTime="3.909346704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:47.373906906 +0000 UTC m=+7.677174557" watchObservedRunningTime="2026-10-31 12:26:47.374662835 +0000 UTC m=+7.677930486"
Oct 31 12:26:47.584269 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:47.584189    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:26:48.250502 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:48.250475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:48.250632 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:48.250616    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:26:48.250989 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:48.250972    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:48.251060 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:48.251047    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:26:48.251204 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:48.251185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:48.251461 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:48.251290    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:26:48.365320 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:48.364829    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9njc" event={"ID":"fe072ecb-08a7-4c60-ad8b-d6643882ba73","Type":"ContainerStarted","Data":"13effb712dcaacd490b9f9d0dd3a8e71c948d3369347e24db12a1d8fed411815"}
Oct 31 12:26:48.367933 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:48.367906    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5" event={"ID":"4be38679-7629-4ec8-aed1-d2ef3fcd8a10","Type":"ContainerStarted","Data":"7cc61fd92053f791cbdbdae1ed892e0c56b2cb0d21b8bed2cf46465e37c61e1d"}
Oct 31 12:26:49.806029 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:49.805466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:49.806029 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:49.805616    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:49.806029 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:49.805675    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret podName:8c53a9b9-23b0-4598-8ee6-35de80dd12d4 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:57.805657016 +0000 UTC m=+18.108924663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret") pod "global-pull-secret-syncer-p8vf7" (UID: "8c53a9b9-23b0-4598-8ee6-35de80dd12d4") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:49.905771 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:49.905741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:49.905892 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:49.905807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l822j\" (UniqueName: \"kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j\") pod \"network-check-target-jhwnt\" (UID: \"23cc7cc6-1673-434b-a56f-e83298dc5d91\") " pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:49.905926 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:49.905911    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:26:49.905926 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:49.905925    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:26:49.905994 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:49.905934    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l822j for pod openshift-network-diagnostics/network-check-target-jhwnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:49.905994 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:49.905982    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j podName:23cc7cc6-1673-434b-a56f-e83298dc5d91 nodeName:}" failed. No retries permitted until 2026-10-31 12:26:57.905966172 +0000 UTC m=+18.209233815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l822j" (UniqueName: "kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j") pod "network-check-target-jhwnt" (UID: "23cc7cc6-1673-434b-a56f-e83298dc5d91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:49.906516 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:49.906494    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:49.906587 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:49.906563    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs podName:95feab9f-adf7-4d34-bde6-d0a1b450df6c nodeName:}" failed. No retries permitted until 2026-10-31 12:26:57.906548036 +0000 UTC m=+18.209815673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs") pod "network-metrics-daemon-vqlp8" (UID: "95feab9f-adf7-4d34-bde6-d0a1b450df6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:49.982006 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:49.981975    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 12:26:50.175064 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.174978    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:26:49.981990401Z","UUID":"2aa7f9e6-8f6c-43d4-9963-2d67469b4151","Handler":null,"Name":"","Endpoint":""}
Oct 31 12:26:50.177482 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.177461    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 12:26:50.177580 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.177513    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 12:26:50.252403 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.251971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:50.252403 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:50.252075    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:26:50.252403 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.252146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:50.252403 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:50.252217    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:26:50.252403 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.252291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:50.252715 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:50.252443    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:26:50.382315 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.382285    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5" event={"ID":"4be38679-7629-4ec8-aed1-d2ef3fcd8a10","Type":"ContainerStarted","Data":"9167a9352c435b394ac36510bd78f77a43ab89af2cd0c35a46a20e7592001d8a"}
Oct 31 12:26:50.384301 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.384277    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2wtl6" event={"ID":"5a733526-fcbe-483b-8e2a-e58c99be5707","Type":"ContainerStarted","Data":"d48e0fb4fd629762c8b11c2f3967b60d9e8e5ecd3b0bbf89fb8e9f48b5bd04c8"}
Oct 31 12:26:50.387098 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.387075    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bw8kq" event={"ID":"ab9d44df-91e2-4e21-afd7-eac98bf106e5","Type":"ContainerStarted","Data":"4699a5f2d9d8e766e2f345d76413149964839ff09b5761fd1ccd44e44540128d"}
Oct 31 12:26:50.420146 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.417757    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s9njc" podStartSLOduration=5.333159031 podStartE2EDuration="10.417738782s" totalImagesPullingTime="5.084579751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:48.381968514 +0000 UTC m=+8.685236164" watchObservedRunningTime="2026-10-31 12:26:50.417738782 +0000 UTC m=+10.721006432"
Oct 31 12:26:50.420146 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.418434    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2wtl6" podStartSLOduration=2.717359103 podStartE2EDuration="10.418424264s" totalImagesPullingTime="7.701065161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:50.417283047 +0000 UTC m=+10.720550699" watchObservedRunningTime="2026-10-31 12:26:50.418424264 +0000 UTC m=+10.721691916"
Oct 31 12:26:50.428142 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.427966    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:26:50.428687 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.428670    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:26:50.433914 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:50.433880    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bw8kq" podStartSLOduration=2.893711201 podStartE2EDuration="10.433868598s" totalImagesPullingTime="7.540157397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:50.433612445 +0000 UTC m=+10.736880099" watchObservedRunningTime="2026-10-31 12:26:50.433868598 +0000 UTC m=+10.737136247"
Oct 31 12:26:51.391388 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:51.391294    2577 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" containerID="7a705325516ff497eb2d443525878988a2e7226cf456979ae44d193e346771b2" exitCode=0
Oct 31 12:26:51.391824 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:51.391389    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lxnjj" event={"ID":"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a","Type":"ContainerDied","Data":"7a705325516ff497eb2d443525878988a2e7226cf456979ae44d193e346771b2"}
Oct 31 12:26:51.393799 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:51.393769    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5" event={"ID":"4be38679-7629-4ec8-aed1-d2ef3fcd8a10","Type":"ContainerStarted","Data":"4bbf9f0910236490fd012c0ddd1187285fdc75866bad2534a3b87e9f0b1b9d1c"}
Oct 31 12:26:51.394896 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:51.394875    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:26:51.452497 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:51.452444    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5" podStartSLOduration=3.215023899 podStartE2EDuration="11.452426107s" totalImagesPullingTime="8.237402208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:51.451761665 +0000 UTC m=+11.755029315" watchObservedRunningTime="2026-10-31 12:26:51.452426107 +0000 UTC m=+11.755693758"
Oct 31 12:26:52.251254 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:52.251011    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:52.251440 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:52.251010    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:52.251440 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:52.251377    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:26:52.251440 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:52.251048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:52.251577 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:52.251447    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:26:52.251577 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:52.251520    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:26:52.398004 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:52.397975    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lxnjj" event={"ID":"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a","Type":"ContainerStarted","Data":"bf449d10fccf61108a1b2547dd6c0b11246077fe6ca818fbf0315bbc3498043c"}
Oct 31 12:26:52.399407 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:52.399337    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5xw6s" event={"ID":"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9","Type":"ContainerStarted","Data":"ac3913fcfe3d0177865f183827ae77518b8b4202be1747fb0def46f9dade7416"}
Oct 31 12:26:52.400817 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:52.400794    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z42qt" event={"ID":"119a3a49-2f9f-4285-bba5-e035d61b03cd","Type":"ContainerStarted","Data":"78ce15fbed23c736991e20af8d9b224d5dd5f7b7856a21c3212c09dc37179352"}
Oct 31 12:26:52.421633 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:52.421590    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5xw6s" podStartSLOduration=2.832832011 podStartE2EDuration="12.421577444s" totalImagesPullingTime="9.588745433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:52.420944686 +0000 UTC m=+12.724212336" watchObservedRunningTime="2026-10-31 12:26:52.421577444 +0000 UTC m=+12.724845094"
Oct 31 12:26:52.437915 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:52.437865    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z42qt" podStartSLOduration=4.488031954 podStartE2EDuration="12.43784832s" totalImagesPullingTime="7.949816366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:26:52.437019552 +0000 UTC m=+12.740287201" watchObservedRunningTime="2026-10-31 12:26:52.43784832 +0000 UTC m=+12.741115974"
Oct 31 12:26:53.405182 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:53.405032    2577 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" containerID="bf449d10fccf61108a1b2547dd6c0b11246077fe6ca818fbf0315bbc3498043c" exitCode=0
Oct 31 12:26:53.405182 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:53.405117    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lxnjj" event={"ID":"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a","Type":"ContainerDied","Data":"bf449d10fccf61108a1b2547dd6c0b11246077fe6ca818fbf0315bbc3498043c"}
Oct 31 12:26:54.250565 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:54.250534    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:54.250730 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:54.250577    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:54.250730 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:54.250544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:54.250730 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:54.250665    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:26:54.250825 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:54.250725    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:26:54.250825 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:54.250811    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:26:54.410057 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:54.410030    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lxnjj" event={"ID":"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a","Type":"ContainerStarted","Data":"3dce64ab0abe72d7eaac81873d34d610fdd1d123af6cf79eba073696bb653f41"}
Oct 31 12:26:55.413666 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:55.413633    2577 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" containerID="3dce64ab0abe72d7eaac81873d34d610fdd1d123af6cf79eba073696bb653f41" exitCode=0
Oct 31 12:26:55.414041 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:55.413691    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lxnjj" event={"ID":"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a","Type":"ContainerDied","Data":"3dce64ab0abe72d7eaac81873d34d610fdd1d123af6cf79eba073696bb653f41"}
Oct 31 12:26:56.251072 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:56.250808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:56.251248 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:56.250808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:56.251248 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:56.251180    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:26:56.251248 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:56.250840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:56.251248 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:56.251231    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:26:56.251495 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:56.251339    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:26:57.867361 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:57.867322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:57.867737 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:57.867491    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:57.867737 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:57.867560    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret podName:8c53a9b9-23b0-4598-8ee6-35de80dd12d4 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:13.867544795 +0000 UTC m=+34.170812428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret") pod "global-pull-secret-syncer-p8vf7" (UID: "8c53a9b9-23b0-4598-8ee6-35de80dd12d4") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:26:57.968309 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:57.968265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l822j\" (UniqueName: \"kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j\") pod \"network-check-target-jhwnt\" (UID: \"23cc7cc6-1673-434b-a56f-e83298dc5d91\") " pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:57.968477 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:57.968345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:57.968547 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:57.968509    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:57.968598 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:57.968571    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs podName:95feab9f-adf7-4d34-bde6-d0a1b450df6c nodeName:}" failed. No retries permitted until 2026-10-31 12:27:13.968553381 +0000 UTC m=+34.271821028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs") pod "network-metrics-daemon-vqlp8" (UID: "95feab9f-adf7-4d34-bde6-d0a1b450df6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:26:57.969021 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:57.969003    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:26:57.969105 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:57.969026    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:26:57.969105 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:57.969040    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l822j for pod openshift-network-diagnostics/network-check-target-jhwnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:57.969105 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:57.969084    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j podName:23cc7cc6-1673-434b-a56f-e83298dc5d91 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:13.969068539 +0000 UTC m=+34.272336181 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l822j" (UniqueName: "kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j") pod "network-check-target-jhwnt" (UID: "23cc7cc6-1673-434b-a56f-e83298dc5d91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:26:58.250582 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:58.250505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:26:58.250730 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:58.250516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:26:58.250730 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:58.250633    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:26:58.250849 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:58.250725    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:26:58.250849 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:58.250777    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:26:58.250849 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:26:58.250832    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:26:58.423234 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:58.423196    2577 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" containerID="098ab671dc814ca2fb8e4b7c297bdef7b8b15853b58fefe094c8c389918ff665" exitCode=0
Oct 31 12:26:58.423438 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:58.423262    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lxnjj" event={"ID":"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a","Type":"ContainerDied","Data":"098ab671dc814ca2fb8e4b7c297bdef7b8b15853b58fefe094c8c389918ff665"}
Oct 31 12:26:59.429217 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:59.429179    2577 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" containerID="b7611865f64259b6d20521cfc16ed795d43c868aa7317d03899b6dd834f4c283" exitCode=0
Oct 31 12:26:59.429703 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:26:59.429244    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lxnjj" event={"ID":"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a","Type":"ContainerDied","Data":"b7611865f64259b6d20521cfc16ed795d43c868aa7317d03899b6dd834f4c283"}
Oct 31 12:27:00.251936 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:00.251762    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:00.252075 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:00.251859    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:00.252143 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:00.251894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:00.252143 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:00.252112    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:27:00.252262 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:00.251998    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:27:00.252262 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:00.252179    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:27:00.433858 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:00.433823    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms" event={"ID":"7287c3ee-7c75-47e2-9c34-d436193f32e3","Type":"ContainerStarted","Data":"cd1a4d80fad238c6a899acaaafadb8221a8a4487703409dc87dc31174bddc470"}
Oct 31 12:27:00.434265 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:00.433909    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms" event={"ID":"7287c3ee-7c75-47e2-9c34-d436193f32e3","Type":"ContainerStarted","Data":"d410d76e2d9468c547002a8fb21c01c1183001ce50d4d80668273fd626dc6a6b"}
Oct 31 12:27:00.434265 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:00.433963    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms" event={"ID":"7287c3ee-7c75-47e2-9c34-d436193f32e3","Type":"ContainerStarted","Data":"6ebf37f01edfa7724b20eab0a2b296bd1644999f1fe1cf34dc50c40654f5f63a"}
Oct 31 12:27:00.434265 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:00.434010    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms" event={"ID":"7287c3ee-7c75-47e2-9c34-d436193f32e3","Type":"ContainerStarted","Data":"de376c66c7eaeccfdaea3ea27df5c8919a159a6fd65c0b1e92fdfaed2829fbf8"}
Oct 31 12:27:00.434265 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:00.434048    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms" event={"ID":"7287c3ee-7c75-47e2-9c34-d436193f32e3","Type":"ContainerStarted","Data":"773874334b6467c35b3c305775441707401d9164531e5e8382977710803e6bfd"}
Oct 31 12:27:00.434265 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:00.434080    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms" event={"ID":"7287c3ee-7c75-47e2-9c34-d436193f32e3","Type":"ContainerStarted","Data":"5f8e8d72976e81f77bbeef85238665db096f1ec958757147b26a9118b9fe6e02"}
Oct 31 12:27:00.436893 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:00.436834    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lxnjj" event={"ID":"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a","Type":"ContainerStarted","Data":"be927c9862559800cfe8de2925563e5b27eccbd695b5dccf3af552f7b0b1be5c"}
Oct 31 12:27:00.462317 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:00.462275    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lxnjj" podStartSLOduration=9.128806297 podStartE2EDuration="20.462263852s" totalImagesPullingTime="11.333457555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:00.462021979 +0000 UTC m=+20.765289629" watchObservedRunningTime="2026-10-31 12:27:00.462263852 +0000 UTC m=+20.765531501"
Oct 31 12:27:02.250730 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:02.250695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:02.251195 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:02.250704    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:02.251195 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:02.250833    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:27:02.251195 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:02.250704    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:02.251195 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:02.250874    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:27:02.251195 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:02.250941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:27:02.444345 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:02.444311    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms" event={"ID":"7287c3ee-7c75-47e2-9c34-d436193f32e3","Type":"ContainerStarted","Data":"2e92ef0573f15efbbe58d9a604918d361a45fc7284d3210e524738635b48c38e"}
Oct 31 12:27:04.251078 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:04.251044    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:04.251694 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:04.251147    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:27:04.251694 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:04.251054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:04.251694 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:04.251061    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:04.251694 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:04.251295    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:27:04.251694 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:04.251347    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:27:04.452548 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:04.452511    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms" event={"ID":"7287c3ee-7c75-47e2-9c34-d436193f32e3","Type":"ContainerStarted","Data":"ab0a66023e811e39ca5dd2ca64fe871d0006406c64ec64ccbb439952c86da7cf"}
Oct 31 12:27:04.452965 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:04.452847    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:04.452965 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:04.452924    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:04.471502 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:04.471472    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:04.471640 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:04.471591    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:04.487827 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:04.487784    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms" podStartSLOduration=7.041152399 podStartE2EDuration="24.487770311s" totalImagesPullingTime="17.446617912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:04.486909354 +0000 UTC m=+24.790177015" watchObservedRunningTime="2026-10-31 12:27:04.487770311 +0000 UTC m=+24.791037958"
Oct 31 12:27:05.454182 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:05.454154    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:27:06.144454 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:06.144422    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p8vf7"]
Oct 31 12:27:06.144586 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:06.144567    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:06.144671 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:06.144652    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:27:06.148447 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:06.148408    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jhwnt"]
Oct 31 12:27:06.148575 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:06.148518    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vqlp8"]
Oct 31 12:27:06.148575 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:06.148526    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:06.148688 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:06.148615    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:27:06.148688 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:06.148673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:06.148768 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:06.148734    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:27:06.456536 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:06.456511    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:27:08.250238 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:08.250202    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:08.250746 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:08.250217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:08.250746 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:08.250316    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:27:08.250746 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:08.250433    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:27:08.250746 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:08.250220    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:08.250746 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:08.250536    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:27:10.251999 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:10.251969    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:10.252416 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:10.252095    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:10.252416 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:10.252110    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:27:10.252416 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:10.252202    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:27:10.252416 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:10.252222    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:10.252416 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:10.252281    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:27:11.770262 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:11.770234    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:11.770748 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:11.770470    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:27:11.789400 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:11.789377    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:12.250840 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:12.250806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:12.250840 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:12.250833    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:12.251076 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:12.250927    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqlp8" podUID="95feab9f-adf7-4d34-bde6-d0a1b450df6c"
Oct 31 12:27:12.251076 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:12.250956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:12.251076 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:12.251006    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8vf7" podUID="8c53a9b9-23b0-4598-8ee6-35de80dd12d4"
Oct 31 12:27:12.251216 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:12.251087    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhwnt" podUID="23cc7cc6-1673-434b-a56f-e83298dc5d91"
Oct 31 12:27:13.056998 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.056973    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeReady"
Oct 31 12:27:13.057500 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.057147    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 12:27:13.107877 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.107848    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7lv7c"]
Oct 31 12:27:13.132597 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.132571    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pjdt2"]
Oct 31 12:27:13.132756 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.132691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7lv7c"
Oct 31 12:27:13.140037 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.140017    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nzzrg\""
Oct 31 12:27:13.140037 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.140027    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 12:27:13.140193 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.140085    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 12:27:13.140193 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.140152    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 12:27:13.145302 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.145287    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7lv7c"]
Oct 31 12:27:13.145348 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.145316    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pjdt2"]
Oct 31 12:27:13.145348 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.145334    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7vcws"]
Oct 31 12:27:13.145423 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.145405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.157865 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.157843    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7vcws"]
Oct 31 12:27:13.157991 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.157966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.240934 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.240906    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 12:27:13.240934 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.240933    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 12:27:13.241160 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.240949    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 12:27:13.241160 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.240976    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c6kzr\""
Oct 31 12:27:13.241160 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.240955    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 12:27:13.241160 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.240954    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 12:27:13.241160 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.241086    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 12:27:13.241160 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.241108    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcqch\""
Oct 31 12:27:13.281601 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.281576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.281711 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.281604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-data-volume\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.281711 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.281632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-crio-socket\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.281711 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.281653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/609830fb-fa9a-4886-89be-cb40635b2c67-cert\") pod \"ingress-canary-7lv7c\" (UID: \"609830fb-fa9a-4886-89be-cb40635b2c67\") " pod="openshift-ingress-canary/ingress-canary-7lv7c"
Oct 31 12:27:13.281868 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.281765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.281922 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.281867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bb3612-89f9-4494-9634-2b8bed367954-tmp-dir\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.281922 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.281896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4j49\" (UniqueName: \"kubernetes.io/projected/f5bb3612-89f9-4494-9634-2b8bed367954-kube-api-access-c4j49\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.281995 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.281921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vpf2\" (UniqueName: \"kubernetes.io/projected/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-api-access-2vpf2\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.281995 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.281958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2shhr\" (UniqueName: \"kubernetes.io/projected/609830fb-fa9a-4886-89be-cb40635b2c67-kube-api-access-2shhr\") pod \"ingress-canary-7lv7c\" (UID: \"609830fb-fa9a-4886-89be-cb40635b2c67\") " pod="openshift-ingress-canary/ingress-canary-7lv7c"
Oct 31 12:27:13.281995 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.281975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb3612-89f9-4494-9634-2b8bed367954-config-volume\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.282082 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.282056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.382850 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.382784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bb3612-89f9-4494-9634-2b8bed367954-tmp-dir\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.382850 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.382815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4j49\" (UniqueName: \"kubernetes.io/projected/f5bb3612-89f9-4494-9634-2b8bed367954-kube-api-access-c4j49\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.382850 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.382836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vpf2\" (UniqueName: \"kubernetes.io/projected/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-api-access-2vpf2\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.383072 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.382853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2shhr\" (UniqueName: \"kubernetes.io/projected/609830fb-fa9a-4886-89be-cb40635b2c67-kube-api-access-2shhr\") pod \"ingress-canary-7lv7c\" (UID: \"609830fb-fa9a-4886-89be-cb40635b2c67\") " pod="openshift-ingress-canary/ingress-canary-7lv7c"
Oct 31 12:27:13.383072 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.382869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb3612-89f9-4494-9634-2b8bed367954-config-volume\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.383072 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.382886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.383072 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.383014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.383072 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.383044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-data-volume\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.383299 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.383082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-crio-socket\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.383299 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.383113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/609830fb-fa9a-4886-89be-cb40635b2c67-cert\") pod \"ingress-canary-7lv7c\" (UID: \"609830fb-fa9a-4886-89be-cb40635b2c67\") " pod="openshift-ingress-canary/ingress-canary-7lv7c"
Oct 31 12:27:13.383299 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.383083    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:13.383299 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.383193    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 12:27:13.383299 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.383203    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bb3612-89f9-4494-9634-2b8bed367954-tmp-dir\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.383299 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.383222    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls podName:adb879bf-aeb5-468c-8f93-4cb40d6e4e36 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:13.883201747 +0000 UTC m=+34.186469390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vcws" (UID: "adb879bf-aeb5-468c-8f93-4cb40d6e4e36") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:13.383299 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.383195    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-crio-socket\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.383299 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.383246    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm podName:adb879bf-aeb5-468c-8f93-4cb40d6e4e36 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:13.883230815 +0000 UTC m=+34.186498456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vcws" (UID: "adb879bf-aeb5-468c-8f93-4cb40d6e4e36") : configmap "kube-rbac-proxy" not found
Oct 31 12:27:13.383299 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.383152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.383299 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.383146    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:13.383668 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.383340    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls podName:f5bb3612-89f9-4494-9634-2b8bed367954 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:13.883322767 +0000 UTC m=+34.186590395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls") pod "dns-default-pjdt2" (UID: "f5bb3612-89f9-4494-9634-2b8bed367954") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:13.383668 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.383393    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-data-volume\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.383668 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.383550    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb3612-89f9-4494-9634-2b8bed367954-config-volume\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.386809 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.386791    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/609830fb-fa9a-4886-89be-cb40635b2c67-cert\") pod \"ingress-canary-7lv7c\" (UID: \"609830fb-fa9a-4886-89be-cb40635b2c67\") " pod="openshift-ingress-canary/ingress-canary-7lv7c"
Oct 31 12:27:13.395459 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.395438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vpf2\" (UniqueName: \"kubernetes.io/projected/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-api-access-2vpf2\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.395587 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.395571    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2shhr\" (UniqueName: \"kubernetes.io/projected/609830fb-fa9a-4886-89be-cb40635b2c67-kube-api-access-2shhr\") pod \"ingress-canary-7lv7c\" (UID: \"609830fb-fa9a-4886-89be-cb40635b2c67\") " pod="openshift-ingress-canary/ingress-canary-7lv7c"
Oct 31 12:27:13.395627 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.395580    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4j49\" (UniqueName: \"kubernetes.io/projected/f5bb3612-89f9-4494-9634-2b8bed367954-kube-api-access-c4j49\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.453074 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.453047    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7lv7c"
Oct 31 12:27:13.792727 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.792500    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7lv7c"]
Oct 31 12:27:13.796705 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:27:13.796676    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609830fb_fa9a_4886_89be_cb40635b2c67.slice/crio-2fc122e5f4ab142e986ebb5a8f5304e189d3616d70df6d9bcdd26cceab90e590 WatchSource:0}: Error finding container 2fc122e5f4ab142e986ebb5a8f5304e189d3616d70df6d9bcdd26cceab90e590: Status 404 returned error can't find the container with id 2fc122e5f4ab142e986ebb5a8f5304e189d3616d70df6d9bcdd26cceab90e590
Oct 31 12:27:13.888160 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.888127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.888330 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.888171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:13.888330 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.888209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:13.888330 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.888259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:13.888330 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.888317    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 12:27:13.888534 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.888343    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:27:13.888534 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.888370    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:13.888534 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.888404    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm podName:adb879bf-aeb5-468c-8f93-4cb40d6e4e36 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:14.888384334 +0000 UTC m=+35.191651963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vcws" (UID: "adb879bf-aeb5-468c-8f93-4cb40d6e4e36") : configmap "kube-rbac-proxy" not found
Oct 31 12:27:13.888534 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.888438    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret podName:8c53a9b9-23b0-4598-8ee6-35de80dd12d4 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:45.888428745 +0000 UTC m=+66.191696376 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret") pod "global-pull-secret-syncer-p8vf7" (UID: "8c53a9b9-23b0-4598-8ee6-35de80dd12d4") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:27:13.888534 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.888369    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:13.888534 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.888449    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls podName:adb879bf-aeb5-468c-8f93-4cb40d6e4e36 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:14.888444232 +0000 UTC m=+35.191711859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vcws" (UID: "adb879bf-aeb5-468c-8f93-4cb40d6e4e36") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:13.888534 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.888505    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls podName:f5bb3612-89f9-4494-9634-2b8bed367954 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:14.888489048 +0000 UTC m=+35.191756690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls") pod "dns-default-pjdt2" (UID: "f5bb3612-89f9-4494-9634-2b8bed367954") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:13.989274 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.989242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l822j\" (UniqueName: \"kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j\") pod \"network-check-target-jhwnt\" (UID: \"23cc7cc6-1673-434b-a56f-e83298dc5d91\") " pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:13.989472 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:13.989297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:13.989472 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.989435    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:27:13.989472 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.989457    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:27:13.989621 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.989532    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs podName:95feab9f-adf7-4d34-bde6-d0a1b450df6c nodeName:}" failed. No retries permitted until 2026-10-31 12:27:45.989510474 +0000 UTC m=+66.292778119 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs") pod "network-metrics-daemon-vqlp8" (UID: "95feab9f-adf7-4d34-bde6-d0a1b450df6c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:27:13.989621 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.989460    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:27:13.989621 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.989560    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l822j for pod openshift-network-diagnostics/network-check-target-jhwnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:27:13.989621 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:13.989607    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j podName:23cc7cc6-1673-434b-a56f-e83298dc5d91 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:45.989591881 +0000 UTC m=+66.292859526 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l822j" (UniqueName: "kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j") pod "network-check-target-jhwnt" (UID: "23cc7cc6-1673-434b-a56f-e83298dc5d91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:27:14.250749 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.250714    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:14.251318 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.250718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:14.251318 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.250852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:14.255616 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.255528    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 12:27:14.255616 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.255559    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 12:27:14.255616 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.255586    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 12:27:14.256592 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.256571    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7td2\""
Oct 31 12:27:14.256592 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.256585    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghjl2\""
Oct 31 12:27:14.256704 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.256663    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mwkk2\""
Oct 31 12:27:14.256704 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.256693    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 12:27:14.474954 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.474924    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bw8kq_ab9d44df-91e2-4e21-afd7-eac98bf106e5/dns-node-resolver/0.log"
Oct 31 12:27:14.477104 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.477080    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7lv7c" event={"ID":"609830fb-fa9a-4886-89be-cb40635b2c67","Type":"ContainerStarted","Data":"2fc122e5f4ab142e986ebb5a8f5304e189d3616d70df6d9bcdd26cceab90e590"}
Oct 31 12:27:14.895295 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.895266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:14.895490 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.895328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:14.895490 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:14.895399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:14.895490 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:14.895429    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:14.895490 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:14.895438    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:14.895490 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:14.895488    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls podName:adb879bf-aeb5-468c-8f93-4cb40d6e4e36 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:16.895474001 +0000 UTC m=+37.198741630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vcws" (UID: "adb879bf-aeb5-468c-8f93-4cb40d6e4e36") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:14.895685 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:14.895494    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 12:27:14.895685 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:14.895502    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls podName:f5bb3612-89f9-4494-9634-2b8bed367954 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:16.89549591 +0000 UTC m=+37.198763539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls") pod "dns-default-pjdt2" (UID: "f5bb3612-89f9-4494-9634-2b8bed367954") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:14.895685 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:14.895540    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm podName:adb879bf-aeb5-468c-8f93-4cb40d6e4e36 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:16.895524463 +0000 UTC m=+37.198792104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vcws" (UID: "adb879bf-aeb5-468c-8f93-4cb40d6e4e36") : configmap "kube-rbac-proxy" not found
Oct 31 12:27:15.646417 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:15.646389    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9njc_fe072ecb-08a7-4c60-ad8b-d6643882ba73/node-ca/0.log"
Oct 31 12:27:16.912324 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:16.912282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:16.912897 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:16.912367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:16.912897 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:16.912421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:16.912897 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:16.912479    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:16.912897 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:16.912519    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 12:27:16.912897 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:16.912559    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls podName:adb879bf-aeb5-468c-8f93-4cb40d6e4e36 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:20.912538038 +0000 UTC m=+41.215805686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vcws" (UID: "adb879bf-aeb5-468c-8f93-4cb40d6e4e36") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:16.912897 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:16.912580    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm podName:adb879bf-aeb5-468c-8f93-4cb40d6e4e36 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:20.912569923 +0000 UTC m=+41.215837555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vcws" (UID: "adb879bf-aeb5-468c-8f93-4cb40d6e4e36") : configmap "kube-rbac-proxy" not found
Oct 31 12:27:16.912897 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:16.912557    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:16.912897 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:16.912613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls podName:f5bb3612-89f9-4494-9634-2b8bed367954 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:20.912605173 +0000 UTC m=+41.215872820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls") pod "dns-default-pjdt2" (UID: "f5bb3612-89f9-4494-9634-2b8bed367954") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:20.495618 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:20.495582    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7lv7c" event={"ID":"609830fb-fa9a-4886-89be-cb40635b2c67","Type":"ContainerStarted","Data":"cf7e18a1213ed8f18366a9450eebddf967eb376fc159883a7c12a5a7d9fd8cb7"}
Oct 31 12:27:20.512786 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:20.512739    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7lv7c" podStartSLOduration=1.375882053 podStartE2EDuration="7.512723981s" totalImagesPullingTime="6.136841928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:20.512581459 +0000 UTC m=+40.815849106" watchObservedRunningTime="2026-10-31 12:27:20.512723981 +0000 UTC m=+40.815991632"
Oct 31 12:27:20.940427 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:20.940391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:20.940599 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:20.940445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:20.940599 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:20.940547    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 12:27:20.940667 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:20.940601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:20.940667 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:20.940624    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls podName:f5bb3612-89f9-4494-9634-2b8bed367954 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:28.940604814 +0000 UTC m=+49.243872454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls") pod "dns-default-pjdt2" (UID: "f5bb3612-89f9-4494-9634-2b8bed367954") : secret "dns-default-metrics-tls" not found
Oct 31 12:27:20.940737 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:20.940677    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:20.940737 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:20.940715    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls podName:adb879bf-aeb5-468c-8f93-4cb40d6e4e36 nodeName:}" failed. No retries permitted until 2026-10-31 12:27:28.940706738 +0000 UTC m=+49.243974389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vcws" (UID: "adb879bf-aeb5-468c-8f93-4cb40d6e4e36") : secret "insights-runtime-extractor-tls" not found
Oct 31 12:27:20.941078 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:20.941061    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:28.998934 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:28.998885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:28.999328 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:28.998973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:29.002253 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:29.002229    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:29.002437 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:29.002411    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:29.094401 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:29.094370    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:29.145673 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:29.145640    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:29.389401 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:29.389367    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pjdt2"]
Oct 31 12:27:29.392959 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:27:29.392930    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bb3612_89f9_4494_9634_2b8bed367954.slice/crio-01a891d83b9d1e5beb4d2163fdec8add534ee2db0c2ffa135000fab8ab56a60a WatchSource:0}: Error finding container 01a891d83b9d1e5beb4d2163fdec8add534ee2db0c2ffa135000fab8ab56a60a: Status 404 returned error can't find the container with id 01a891d83b9d1e5beb4d2163fdec8add534ee2db0c2ffa135000fab8ab56a60a
Oct 31 12:27:29.419878 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:29.419849    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7vcws"]
Oct 31 12:27:29.422708 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:27:29.422686    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb879bf_aeb5_468c_8f93_4cb40d6e4e36.slice/crio-5dbf68e07df4d2fd91d150bbff7ff5cb9c41d281ddb536ab6c39c475e9af0408 WatchSource:0}: Error finding container 5dbf68e07df4d2fd91d150bbff7ff5cb9c41d281ddb536ab6c39c475e9af0408: Status 404 returned error can't find the container with id 5dbf68e07df4d2fd91d150bbff7ff5cb9c41d281ddb536ab6c39c475e9af0408
Oct 31 12:27:29.519389 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:29.519346    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vcws" event={"ID":"adb879bf-aeb5-468c-8f93-4cb40d6e4e36","Type":"ContainerStarted","Data":"b06fed73c2a06d45a8889418fcae74660387579f9a601d36cbadb653d9ef8d96"}
Oct 31 12:27:29.519529 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:29.519416    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vcws" event={"ID":"adb879bf-aeb5-468c-8f93-4cb40d6e4e36","Type":"ContainerStarted","Data":"5dbf68e07df4d2fd91d150bbff7ff5cb9c41d281ddb536ab6c39c475e9af0408"}
Oct 31 12:27:29.520475 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:29.520451    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pjdt2" event={"ID":"f5bb3612-89f9-4494-9634-2b8bed367954","Type":"ContainerStarted","Data":"01a891d83b9d1e5beb4d2163fdec8add534ee2db0c2ffa135000fab8ab56a60a"}
Oct 31 12:27:30.524591 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.524513    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vcws" event={"ID":"adb879bf-aeb5-468c-8f93-4cb40d6e4e36","Type":"ContainerStarted","Data":"281307ada6b5a2cd14c5d08c8c7be250bdb61a7f4944046ce42c6010f120c11c"}
Oct 31 12:27:30.876450 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.876414    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"]
Oct 31 12:27:30.878770 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.878741    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"
Oct 31 12:27:30.896326 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.896303    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-llbgd\""
Oct 31 12:27:30.909493 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.909464    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"]
Oct 31 12:27:30.969562 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.969521    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-j5n9s"]
Oct 31 12:27:30.971322 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.971296    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b6467d66c-vrqf6"]
Oct 31 12:27:30.971469 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.971441    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:27:30.973067 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.973050    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:30.979971 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.979560    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 12:27:30.979971 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.979587    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 12:27:30.985059 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.985003    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j5n9s"]
Oct 31 12:27:30.989325 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.989266    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b6467d66c-vrqf6"]
Oct 31 12:27:30.998062 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:30.998035    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 12:27:31.002145 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.002121    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 12:27:31.002271 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.002173    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 12:27:31.002271 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.002177    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ght96\""
Oct 31 12:27:31.002271 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.002205    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 12:27:31.002271 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.002237    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-jl5lt\""
Oct 31 12:27:31.002646 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.002632    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 12:27:31.012451 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.012419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/542d428b-b94d-4033-9805-d13da84ede85-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qzkwf\" (UID: \"542d428b-b94d-4033-9805-d13da84ede85\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"
Oct 31 12:27:31.113463 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.113429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be73966c-8b20-4130-8203-5cf6ca09df27-installation-pull-secrets\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.113463 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.113472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-registry-tls\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.113705 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.113494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be73966c-8b20-4130-8203-5cf6ca09df27-ca-trust-extracted\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.113705 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.113603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026-tmp\") pod \"downloads-6dd789c8b7-j5n9s\" (UID: \"5eb94f91-2be3-4a36-8d36-b3fdfb3bc026\") " pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:27:31.113705 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.113691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be73966c-8b20-4130-8203-5cf6ca09df27-registry-certificates\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.113852 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.113727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-bound-sa-token\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.113852 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.113785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/542d428b-b94d-4033-9805-d13da84ede85-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qzkwf\" (UID: \"542d428b-b94d-4033-9805-d13da84ede85\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"
Oct 31 12:27:31.113852 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.113817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be73966c-8b20-4130-8203-5cf6ca09df27-image-registry-private-configuration\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.113988 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.113849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbvh\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-kube-api-access-zsbvh\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.113988 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.113907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzvjn\" (UniqueName: \"kubernetes.io/projected/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026-kube-api-access-nzvjn\") pod \"downloads-6dd789c8b7-j5n9s\" (UID: \"5eb94f91-2be3-4a36-8d36-b3fdfb3bc026\") " pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:27:31.113988 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.113939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be73966c-8b20-4130-8203-5cf6ca09df27-trusted-ca\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.116975 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.116946    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/542d428b-b94d-4033-9805-d13da84ede85-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qzkwf\" (UID: \"542d428b-b94d-4033-9805-d13da84ede85\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"
Oct 31 12:27:31.195450 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.195377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"
Oct 31 12:27:31.214788 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.214756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be73966c-8b20-4130-8203-5cf6ca09df27-image-registry-private-configuration\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.214918 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.214803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbvh\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-kube-api-access-zsbvh\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.214918 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.214861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzvjn\" (UniqueName: \"kubernetes.io/projected/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026-kube-api-access-nzvjn\") pod \"downloads-6dd789c8b7-j5n9s\" (UID: \"5eb94f91-2be3-4a36-8d36-b3fdfb3bc026\") " pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:27:31.214918 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.214893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be73966c-8b20-4130-8203-5cf6ca09df27-trusted-ca\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.215078 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.214920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be73966c-8b20-4130-8203-5cf6ca09df27-installation-pull-secrets\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.215078 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.214948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-registry-tls\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.215078 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.214970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be73966c-8b20-4130-8203-5cf6ca09df27-ca-trust-extracted\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.215078 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.215009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026-tmp\") pod \"downloads-6dd789c8b7-j5n9s\" (UID: \"5eb94f91-2be3-4a36-8d36-b3fdfb3bc026\") " pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:27:31.215078 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.215060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be73966c-8b20-4130-8203-5cf6ca09df27-registry-certificates\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.215315 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.215083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-bound-sa-token\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.216823 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.216111    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026-tmp\") pod \"downloads-6dd789c8b7-j5n9s\" (UID: \"5eb94f91-2be3-4a36-8d36-b3fdfb3bc026\") " pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:27:31.216823 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.216339    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be73966c-8b20-4130-8203-5cf6ca09df27-ca-trust-extracted\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.217012 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.216991    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be73966c-8b20-4130-8203-5cf6ca09df27-registry-certificates\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.217775 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.217722    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be73966c-8b20-4130-8203-5cf6ca09df27-trusted-ca\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.219507 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.219477    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be73966c-8b20-4130-8203-5cf6ca09df27-installation-pull-secrets\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.219714 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.219693    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-registry-tls\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.219762 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.219710    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be73966c-8b20-4130-8203-5cf6ca09df27-image-registry-private-configuration\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.229462 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.229438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-bound-sa-token\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.229865 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.229846    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbvh\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-kube-api-access-zsbvh\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.229982 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.229965    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzvjn\" (UniqueName: \"kubernetes.io/projected/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026-kube-api-access-nzvjn\") pod \"downloads-6dd789c8b7-j5n9s\" (UID: \"5eb94f91-2be3-4a36-8d36-b3fdfb3bc026\") " pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:27:31.286569 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.286539    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:27:31.299186 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.299167    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:31.470219 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.470195    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"]
Oct 31 12:27:31.473102 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:27:31.473072    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542d428b_b94d_4033_9805_d13da84ede85.slice/crio-d16185dc0451c4f52bf242e824c1beefc35ba756c3b32bfbd8a6e3a3906fd6eb WatchSource:0}: Error finding container d16185dc0451c4f52bf242e824c1beefc35ba756c3b32bfbd8a6e3a3906fd6eb: Status 404 returned error can't find the container with id d16185dc0451c4f52bf242e824c1beefc35ba756c3b32bfbd8a6e3a3906fd6eb
Oct 31 12:27:31.527917 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.527882    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf" event={"ID":"542d428b-b94d-4033-9805-d13da84ede85","Type":"ContainerStarted","Data":"d16185dc0451c4f52bf242e824c1beefc35ba756c3b32bfbd8a6e3a3906fd6eb"}
Oct 31 12:27:31.558084 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.558052    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j5n9s"]
Oct 31 12:27:31.560737 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:27:31.560709    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb94f91_2be3_4a36_8d36_b3fdfb3bc026.slice/crio-fb8772d40acb2f2af7233e333108df2a91d1cd8b56a70e62945993c76bd75a32 WatchSource:0}: Error finding container fb8772d40acb2f2af7233e333108df2a91d1cd8b56a70e62945993c76bd75a32: Status 404 returned error can't find the container with id fb8772d40acb2f2af7233e333108df2a91d1cd8b56a70e62945993c76bd75a32
Oct 31 12:27:31.588606 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:31.588580    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b6467d66c-vrqf6"]
Oct 31 12:27:31.592392 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:27:31.592365    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe73966c_8b20_4130_8203_5cf6ca09df27.slice/crio-ab06aedaec96eebe2acde541f69eef57a4330288fe1b90f919e2f183a386e916 WatchSource:0}: Error finding container ab06aedaec96eebe2acde541f69eef57a4330288fe1b90f919e2f183a386e916: Status 404 returned error can't find the container with id ab06aedaec96eebe2acde541f69eef57a4330288fe1b90f919e2f183a386e916
Oct 31 12:27:32.532815 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:32.532722    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vcws" event={"ID":"adb879bf-aeb5-468c-8f93-4cb40d6e4e36","Type":"ContainerStarted","Data":"084950d99871918209ca06b91ded59f4364a59e2bb3a515cdb4900aa6e159922"}
Oct 31 12:27:32.534628 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:32.534595    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b6467d66c-vrqf6" event={"ID":"be73966c-8b20-4130-8203-5cf6ca09df27","Type":"ContainerStarted","Data":"1b5d17e443b309482aafa091a04da38f796094e6d36d267dd2f5b86e88b3ff34"}
Oct 31 12:27:32.534761 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:32.534645    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b6467d66c-vrqf6" event={"ID":"be73966c-8b20-4130-8203-5cf6ca09df27","Type":"ContainerStarted","Data":"ab06aedaec96eebe2acde541f69eef57a4330288fe1b90f919e2f183a386e916"}
Oct 31 12:27:32.534761 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:32.534704    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:32.536133 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:32.536107    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j5n9s" event={"ID":"5eb94f91-2be3-4a36-8d36-b3fdfb3bc026","Type":"ContainerStarted","Data":"fb8772d40acb2f2af7233e333108df2a91d1cd8b56a70e62945993c76bd75a32"}
Oct 31 12:27:32.554043 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:32.554002    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7vcws" podStartSLOduration=16.948212055 podStartE2EDuration="19.553991267s" totalImagesPullingTime="2.605779212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:27:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:32.552969347 +0000 UTC m=+52.856236997" watchObservedRunningTime="2026-10-31 12:27:32.553991267 +0000 UTC m=+52.857258917"
Oct 31 12:27:32.575408 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:32.575371    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b6467d66c-vrqf6" podStartSLOduration=2.575347546 podStartE2EDuration="2.575347546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:32.574978541 +0000 UTC m=+52.878246190" watchObservedRunningTime="2026-10-31 12:27:32.575347546 +0000 UTC m=+52.878615190"
Oct 31 12:27:33.055635 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.055609    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bw8kq_ab9d44df-91e2-4e21-afd7-eac98bf106e5/dns-node-resolver/0.log"
Oct 31 12:27:33.539804 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.539763    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf" event={"ID":"542d428b-b94d-4033-9805-d13da84ede85","Type":"ContainerStarted","Data":"8b57056f4dc4ae9bda2444a444f879a03d3d2cfffedf9315a10fa9f0f5dcc920"}
Oct 31 12:27:33.540263 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.540220    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"
Oct 31 12:27:33.546774 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.546744    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"
Oct 31 12:27:33.561319 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.561264    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf" podStartSLOduration=2.039689348 podStartE2EDuration="3.56124968s" totalImagesPullingTime="1.521560332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:33.560902633 +0000 UTC m=+53.864170284" watchObservedRunningTime="2026-10-31 12:27:33.56124968 +0000 UTC m=+53.864517331"
Oct 31 12:27:33.960268 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.960228    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"]
Oct 31 12:27:33.962430 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.962397    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:33.968111 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.968066    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 12:27:33.968331 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.968309    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 12:27:33.968456 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.968389    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 12:27:33.968456 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.968400    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bpkkl\""
Oct 31 12:27:33.968456 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.968404    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 12:27:33.968605 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.968494    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 12:27:33.973114 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:33.973085    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"]
Oct 31 12:27:34.038947 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.038893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3ba234a-bb37-4ed4-a505-4a8494d4e862-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.039120 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.038954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5tbp\" (UniqueName: \"kubernetes.io/projected/a3ba234a-bb37-4ed4-a505-4a8494d4e862-kube-api-access-l5tbp\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.039120 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.039018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3ba234a-bb37-4ed4-a505-4a8494d4e862-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.039120 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.039063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3ba234a-bb37-4ed4-a505-4a8494d4e862-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.056788 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.056760    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b6467d66c-vrqf6_be73966c-8b20-4130-8203-5cf6ca09df27/registry/0.log"
Oct 31 12:27:34.140185 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.140149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3ba234a-bb37-4ed4-a505-4a8494d4e862-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.140377 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.140247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3ba234a-bb37-4ed4-a505-4a8494d4e862-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.140377 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.140278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5tbp\" (UniqueName: \"kubernetes.io/projected/a3ba234a-bb37-4ed4-a505-4a8494d4e862-kube-api-access-l5tbp\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.140377 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.140312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3ba234a-bb37-4ed4-a505-4a8494d4e862-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.141385 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.141016    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3ba234a-bb37-4ed4-a505-4a8494d4e862-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.143053 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.143030    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3ba234a-bb37-4ed4-a505-4a8494d4e862-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.143386 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.143343    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3ba234a-bb37-4ed4-a505-4a8494d4e862-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.152766 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.152745    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5tbp\" (UniqueName: \"kubernetes.io/projected/a3ba234a-bb37-4ed4-a505-4a8494d4e862-kube-api-access-l5tbp\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.276439 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.276344    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:34.457224 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.456992    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9njc_fe072ecb-08a7-4c60-ad8b-d6643882ba73/node-ca/0.log"
Oct 31 12:27:34.553390 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:34.553302    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"]
Oct 31 12:27:34.556571 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:27:34.556545    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ba234a_bb37_4ed4_a505_4a8494d4e862.slice/crio-08e7cb028de0cc208ae8b278f08edc9368fa95b759a296047045c847c4f0d33f WatchSource:0}: Error finding container 08e7cb028de0cc208ae8b278f08edc9368fa95b759a296047045c847c4f0d33f: Status 404 returned error can't find the container with id 08e7cb028de0cc208ae8b278f08edc9368fa95b759a296047045c847c4f0d33f
Oct 31 12:27:35.056753 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:35.056720    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7lv7c_609830fb-fa9a-4886-89be-cb40635b2c67/serve-healthcheck-canary/0.log"
Oct 31 12:27:35.547368 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:35.547268    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z" event={"ID":"a3ba234a-bb37-4ed4-a505-4a8494d4e862","Type":"ContainerStarted","Data":"08e7cb028de0cc208ae8b278f08edc9368fa95b759a296047045c847c4f0d33f"}
Oct 31 12:27:35.549269 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:35.549241    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pjdt2" event={"ID":"f5bb3612-89f9-4494-9634-2b8bed367954","Type":"ContainerStarted","Data":"2aeb3487e30c35cb82d43357324a14b387c7a60b732e16408f44fd2a55275fbb"}
Oct 31 12:27:35.549413 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:35.549288    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pjdt2" event={"ID":"f5bb3612-89f9-4494-9634-2b8bed367954","Type":"ContainerStarted","Data":"f6d44e04a160ef5b882155ba73eab082c656fdb174cfd856b3f3bca5df54b204"}
Oct 31 12:27:35.570194 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:35.570138    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-pjdt2" podStartSLOduration=17.369139679 podStartE2EDuration="22.570122221s" totalImagesPullingTime="5.200982542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:35.568239137 +0000 UTC m=+55.871506788" watchObservedRunningTime="2026-10-31 12:27:35.570122221 +0000 UTC m=+55.873389873"
Oct 31 12:27:36.556247 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:36.553454    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:37.558593 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:37.558559    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z" event={"ID":"a3ba234a-bb37-4ed4-a505-4a8494d4e862","Type":"ContainerStarted","Data":"4e7ebb60aa886218784616f5f44b5c4b7235320ebc6e55fbfa59d92769eeafbe"}
Oct 31 12:27:37.559091 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:37.558617    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z" event={"ID":"a3ba234a-bb37-4ed4-a505-4a8494d4e862","Type":"ContainerStarted","Data":"14dd00602139e6921b88c76342d7874f55e165018040efd491d6673e87f973cf"}
Oct 31 12:27:37.578543 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:37.578491    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z" podStartSLOduration=2.530168243 podStartE2EDuration="4.578477659s" totalImagesPullingTime="2.048309416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:37.577164879 +0000 UTC m=+57.880432542" watchObservedRunningTime="2026-10-31 12:27:37.578477659 +0000 UTC m=+57.881745308"
Oct 31 12:27:39.360030 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.359997    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2rpgc"]
Oct 31 12:27:39.363524 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.363499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.369441 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.369418    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 12:27:39.370664 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.370640    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9c5q9\""
Oct 31 12:27:39.372202 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.372180    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 12:27:39.383271 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.383244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-wtmp\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.383418 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.383285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.383484 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.383462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-tls\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.383545 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.383502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0589f9a7-c2b8-408f-8374-a25b489ba69a-metrics-client-ca\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.383601 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.383547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wvqq\" (UniqueName: \"kubernetes.io/projected/0589f9a7-c2b8-408f-8374-a25b489ba69a-kube-api-access-8wvqq\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.383669 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.383626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-sys\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.383719 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.383667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.383719 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.383705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-root\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.383812 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.383726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-textfile\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.450770 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.450736    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 12:27:39.485109 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-root\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485109 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-textfile\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485285 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-wtmp\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485285 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485285 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485187    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-root\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485285 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-tls\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485285 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0589f9a7-c2b8-408f-8374-a25b489ba69a-metrics-client-ca\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485285 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvqq\" (UniqueName: \"kubernetes.io/projected/0589f9a7-c2b8-408f-8374-a25b489ba69a-kube-api-access-8wvqq\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485576 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:39.485303    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Oct 31 12:27:39.485576 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-sys\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485576 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485576 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485367    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-wtmp\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485576 ip-10-0-140-137 kubenswrapper[2577]: E1031 12:27:39.485389    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-tls podName:0589f9a7-c2b8-408f-8374-a25b489ba69a nodeName:}" failed. No retries permitted until 2026-10-31 12:27:39.985343754 +0000 UTC m=+60.288611397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-tls") pod "node-exporter-2rpgc" (UID: "0589f9a7-c2b8-408f-8374-a25b489ba69a") : secret "node-exporter-tls" not found
Oct 31 12:27:39.485576 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485416    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-sys\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485576 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485477    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-textfile\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485848 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485776    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0589f9a7-c2b8-408f-8374-a25b489ba69a-metrics-client-ca\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.485881 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.485850    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.487511 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.487490    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.501398 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.501379    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvqq\" (UniqueName: \"kubernetes.io/projected/0589f9a7-c2b8-408f-8374-a25b489ba69a-kube-api-access-8wvqq\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.990426 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.990389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-tls\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:39.992850 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:39.992817    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-tls\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:40.281747 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:40.281666    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9c5q9\""
Oct 31 12:27:40.289921 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:40.289893    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:40.303705 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:27:40.301422    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0589f9a7_c2b8_408f_8374_a25b489ba69a.slice/crio-6de405225e6586537bf4bb3fd5fd4d77051b5ad74447bedbda16bdd625aebd65 WatchSource:0}: Error finding container 6de405225e6586537bf4bb3fd5fd4d77051b5ad74447bedbda16bdd625aebd65: Status 404 returned error can't find the container with id 6de405225e6586537bf4bb3fd5fd4d77051b5ad74447bedbda16bdd625aebd65
Oct 31 12:27:40.572557 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:40.572516    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rpgc" event={"ID":"0589f9a7-c2b8-408f-8374-a25b489ba69a","Type":"ContainerStarted","Data":"6de405225e6586537bf4bb3fd5fd4d77051b5ad74447bedbda16bdd625aebd65"}
Oct 31 12:27:42.580363 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:42.580326    2577 generic.go:356] "Generic (PLEG): container finished" podID="0589f9a7-c2b8-408f-8374-a25b489ba69a" containerID="7183efda5d50ebd0b5dbd156260a12e05b67bc3a6f6424fa4a75387800fee765" exitCode=0
Oct 31 12:27:42.580773 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:42.580419    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rpgc" event={"ID":"0589f9a7-c2b8-408f-8374-a25b489ba69a","Type":"ContainerDied","Data":"7183efda5d50ebd0b5dbd156260a12e05b67bc3a6f6424fa4a75387800fee765"}
Oct 31 12:27:43.587022 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.586986    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rpgc" event={"ID":"0589f9a7-c2b8-408f-8374-a25b489ba69a","Type":"ContainerStarted","Data":"77ec2671326ba7bd803104f050c4a88bcacfaa3f24912b5d557a6f7b89b120a1"}
Oct 31 12:27:43.587506 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.587043    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rpgc" event={"ID":"0589f9a7-c2b8-408f-8374-a25b489ba69a","Type":"ContainerStarted","Data":"0e3f3b45ecaf66add02ba87a3f62b2e84d4427881975efd1ea8544feb71b37ac"}
Oct 31 12:27:43.610558 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.610503    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2rpgc" podStartSLOduration=3.218012864 podStartE2EDuration="4.610488588s" totalImagesPullingTime="1.323406702s" totalInitContainerRuntime="69.069022ms" isStatefulPod=true podCreationTimestamp="2026-10-31 12:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:43.608863828 +0000 UTC m=+63.912131478" watchObservedRunningTime="2026-10-31 12:27:43.610488588 +0000 UTC m=+63.913756238"
Oct 31 12:27:43.757211 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.757177    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-647d9d5db7-nxnt2"]
Oct 31 12:27:43.759050 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.759029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.762424 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.762402    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 12:27:43.762504 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.762456    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 12:27:43.770730 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.770710    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-647d9d5db7-nxnt2"]
Oct 31 12:27:43.796986 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.796960    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 12:27:43.797115 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.797029    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9msvofqo3uh0k\""
Oct 31 12:27:43.797115 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.797052    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qx5pt\""
Oct 31 12:27:43.797115 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.797080    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 12:27:43.824831 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.824796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.825000 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.824878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-secret-metrics-server-client-certs\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.825070 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.825014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmtfl\" (UniqueName: \"kubernetes.io/projected/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-kube-api-access-gmtfl\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.825070 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.825060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-metrics-server-audit-profiles\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.825167 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.825089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-client-ca-bundle\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.825167 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.825135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-audit-log\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.825273 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.825169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-secret-metrics-server-tls\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.925891 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.925809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-audit-log\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.925891 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.925855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-secret-metrics-server-tls\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.926105 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.925897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.926105 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.925931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-secret-metrics-server-client-certs\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.926105 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.925971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmtfl\" (UniqueName: \"kubernetes.io/projected/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-kube-api-access-gmtfl\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.926105 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.926003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-metrics-server-audit-profiles\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.926105 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.926024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-client-ca-bundle\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.926344 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.926238    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-audit-log\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.926770 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.926743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.927092 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.927071    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-metrics-server-audit-profiles\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.928867 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.928841    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-client-ca-bundle\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.928867 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.928853    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-secret-metrics-server-tls\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.929017 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.928922    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-secret-metrics-server-client-certs\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:43.938442 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:43.938422    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmtfl\" (UniqueName: \"kubernetes.io/projected/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-kube-api-access-gmtfl\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:44.070598 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:44.070554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:44.357114 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:44.357080    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-647d9d5db7-nxnt2"]
Oct 31 12:27:44.362636 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:27:44.362601    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1507e7d6_a6e4_4cb9_adf5_c24d8619cede.slice/crio-1749c085df22cc8d63e2372be0379fde981c06b1ea93704510a25f6359689ee2 WatchSource:0}: Error finding container 1749c085df22cc8d63e2372be0379fde981c06b1ea93704510a25f6359689ee2: Status 404 returned error can't find the container with id 1749c085df22cc8d63e2372be0379fde981c06b1ea93704510a25f6359689ee2
Oct 31 12:27:44.591545 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:44.591510    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2" event={"ID":"1507e7d6-a6e4-4cb9-adf5-c24d8619cede","Type":"ContainerStarted","Data":"1749c085df22cc8d63e2372be0379fde981c06b1ea93704510a25f6359689ee2"}
Oct 31 12:27:45.946042 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:45.946004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:45.948688 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:45.948667    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 12:27:45.960490 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:45.960459    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:46.047776 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.047735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l822j\" (UniqueName: \"kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j\") pod \"network-check-target-jhwnt\" (UID: \"23cc7cc6-1673-434b-a56f-e83298dc5d91\") " pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:46.047970 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.047811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:46.050766 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.050741    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 12:27:46.050967 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.050950    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 12:27:46.061464 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.061419    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 12:27:46.063035 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.063008    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:46.073128 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.073098    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l822j\" (UniqueName: \"kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j\") pod \"network-check-target-jhwnt\" (UID: \"23cc7cc6-1673-434b-a56f-e83298dc5d91\") " pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:46.138369 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.138324    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7td2\""
Oct 31 12:27:46.144997 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.144957    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mwkk2\""
Oct 31 12:27:46.146037 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.146015    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:46.153427 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.153400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:46.157637 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.157607    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghjl2\""
Oct 31 12:27:46.164786 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.164726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:46.437805 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.437771    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vqlp8"]
Oct 31 12:27:46.440044 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.440020    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p8vf7"]
Oct 31 12:27:46.443798 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:27:46.443414    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c53a9b9_23b0_4598_8ee6_35de80dd12d4.slice/crio-95ebfde83dc014dfcdd70f0a0603335cd77ac6fc0b1b1ca9342a3535e5722674 WatchSource:0}: Error finding container 95ebfde83dc014dfcdd70f0a0603335cd77ac6fc0b1b1ca9342a3535e5722674: Status 404 returned error can't find the container with id 95ebfde83dc014dfcdd70f0a0603335cd77ac6fc0b1b1ca9342a3535e5722674
Oct 31 12:27:46.466096 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.466023    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jhwnt"]
Oct 31 12:27:46.469367 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:27:46.469327    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cc7cc6_1673_434b_a56f_e83298dc5d91.slice/crio-9d400fd1ede94d33680989708eec336060c5b70f64cf2ddd8895a4bcf58b7b83 WatchSource:0}: Error finding container 9d400fd1ede94d33680989708eec336060c5b70f64cf2ddd8895a4bcf58b7b83: Status 404 returned error can't find the container with id 9d400fd1ede94d33680989708eec336060c5b70f64cf2ddd8895a4bcf58b7b83
Oct 31 12:27:46.560872 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.560839    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:46.607736 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.607700    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p8vf7" event={"ID":"8c53a9b9-23b0-4598-8ee6-35de80dd12d4","Type":"ContainerStarted","Data":"95ebfde83dc014dfcdd70f0a0603335cd77ac6fc0b1b1ca9342a3535e5722674"}
Oct 31 12:27:46.609106 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.609079    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vqlp8" event={"ID":"95feab9f-adf7-4d34-bde6-d0a1b450df6c","Type":"ContainerStarted","Data":"d096c0ea1eaf621fa65e102e73723edbc2373d4feb522719fb85712a6c3450c7"}
Oct 31 12:27:46.610316 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:46.610295    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jhwnt" event={"ID":"23cc7cc6-1673-434b-a56f-e83298dc5d91","Type":"ContainerStarted","Data":"9d400fd1ede94d33680989708eec336060c5b70f64cf2ddd8895a4bcf58b7b83"}
Oct 31 12:27:47.616702 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:47.615865    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2" event={"ID":"1507e7d6-a6e4-4cb9-adf5-c24d8619cede","Type":"ContainerStarted","Data":"8ad329e2af4ab229d9663e8b91eb95cb850de40332437b8a03c3b827edb20cfe"}
Oct 31 12:27:47.639269 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:47.638655    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2" podStartSLOduration=2.08900409 podStartE2EDuration="4.638641067s" totalImagesPullingTime="2.549636977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:47.637696337 +0000 UTC m=+67.940963986" watchObservedRunningTime="2026-10-31 12:27:47.638641067 +0000 UTC m=+67.941908717"
Oct 31 12:27:48.621271 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:48.621233    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vqlp8" event={"ID":"95feab9f-adf7-4d34-bde6-d0a1b450df6c","Type":"ContainerStarted","Data":"e4e5d90ad1e2f62c59ef93e3bd598461213bb394ff088f4e6ebe70ad3d04c928"}
Oct 31 12:27:48.621760 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:48.621294    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vqlp8" event={"ID":"95feab9f-adf7-4d34-bde6-d0a1b450df6c","Type":"ContainerStarted","Data":"d53e23c0148fe51c687139412fdac42e3f7dc0d1014f747c4175030219b78e98"}
Oct 31 12:27:48.640205 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:48.640147    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vqlp8" podStartSLOduration=67.167221196 podStartE2EDuration="1m8.640132975s" totalImagesPullingTime="1.472911779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:48.638737798 +0000 UTC m=+68.942005449" watchObservedRunningTime="2026-10-31 12:27:48.640132975 +0000 UTC m=+68.943400625"
Oct 31 12:27:51.057703 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.057657    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-65d5579bc6-rhzg5"]
Oct 31 12:27:51.062654 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.062629    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.066234 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.066090    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 12:27:51.066234 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.066153    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 12:27:51.066234 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.066205    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 12:27:51.070525 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.070500    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 12:27:51.073720 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.073260    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65d5579bc6-rhzg5"]
Oct 31 12:27:51.105285 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.105256    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 12:27:51.105456 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.105311    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 12:27:51.105456 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.105345    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9nc4h\""
Oct 31 12:27:51.195414 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.195376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-serving-cert\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.195590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.195435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-oauth-config\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.195590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.195464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-trusted-ca-bundle\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.195590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.195489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhf7n\" (UniqueName: \"kubernetes.io/projected/59c1bd59-6c93-426a-9e81-a410a9cc62f4-kube-api-access-fhf7n\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.195590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.195528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-oauth-serving-cert\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.195590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.195552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c1bd59-6c93-426a-9e81-a410a9cc62f4-tmp\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.195590 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.195583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-config\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.195828 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.195623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-service-ca\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.296230 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.296191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-oauth-config\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.296419 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.296241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-trusted-ca-bundle\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.296419 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.296268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhf7n\" (UniqueName: \"kubernetes.io/projected/59c1bd59-6c93-426a-9e81-a410a9cc62f4-kube-api-access-fhf7n\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.296419 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.296311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-oauth-serving-cert\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.296419 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.296338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c1bd59-6c93-426a-9e81-a410a9cc62f4-tmp\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.296419 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.296387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-config\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.296632 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.296428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-service-ca\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.296632 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.296497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-serving-cert\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.297579 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.297547    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c1bd59-6c93-426a-9e81-a410a9cc62f4-tmp\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.297710 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.297686    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-oauth-serving-cert\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.297769 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.297739    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-config\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.297963 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.297938    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-trusted-ca-bundle\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.298253 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.298231    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-service-ca\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.299553 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.299530    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-serving-cert\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.307119 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.307072    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-oauth-config\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.314459 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.314395    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhf7n\" (UniqueName: \"kubernetes.io/projected/59c1bd59-6c93-426a-9e81-a410a9cc62f4-kube-api-access-fhf7n\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.379306 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.379269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:51.672251 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:51.672096    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65d5579bc6-rhzg5"]
Oct 31 12:27:51.677246 ip-10-0-140-137 kubenswrapper[2577]: W1031 12:27:51.676949    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c1bd59_6c93_426a_9e81_a410a9cc62f4.slice/crio-c5b8b9b880947cf81957233c5d94ee1156d415144bf00c0eb1eb667c7949e657 WatchSource:0}: Error finding container c5b8b9b880947cf81957233c5d94ee1156d415144bf00c0eb1eb667c7949e657: Status 404 returned error can't find the container with id c5b8b9b880947cf81957233c5d94ee1156d415144bf00c0eb1eb667c7949e657
Oct 31 12:27:52.641003 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:52.640502    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jhwnt" event={"ID":"23cc7cc6-1673-434b-a56f-e83298dc5d91","Type":"ContainerStarted","Data":"021bb6c1cea4d144af7cdcb6aa0f75a1cbcf9b365ad7e11239aed0eaa8076550"}
Oct 31 12:27:52.641003 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:52.640653    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:52.642789 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:52.642761    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d5579bc6-rhzg5" event={"ID":"59c1bd59-6c93-426a-9e81-a410a9cc62f4","Type":"ContainerStarted","Data":"c5b8b9b880947cf81957233c5d94ee1156d415144bf00c0eb1eb667c7949e657"}
Oct 31 12:27:52.658263 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:52.658207    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jhwnt" podStartSLOduration=66.683760952 podStartE2EDuration="1m12.658192652s" totalImagesPullingTime="5.9744317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:27:52.657938494 +0000 UTC m=+72.961206145" watchObservedRunningTime="2026-10-31 12:27:52.658192652 +0000 UTC m=+72.961460307"
Oct 31 12:27:53.546908 ip-10-0-140-137 kubenswrapper[2577]: I1031 12:27:53.546884    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:55.181207 ip-10-0-140-137 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 12:27:55.199846 ip-10-0-140-137 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 12:27:55.200035 ip-10-0-140-137 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 12:27:55.200230 ip-10-0-140-137 systemd[1]: kubelet.service: Consumed 3.756s CPU time.
Oct 31 12:27:55.211128 ip-10-0-140-137 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 12:27:55.301277 ip-10-0-140-137 kubenswrapper[5663]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:27:55.301277 ip-10-0-140-137 kubenswrapper[5663]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 12:27:55.301277 ip-10-0-140-137 kubenswrapper[5663]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:27:55.301277 ip-10-0-140-137 kubenswrapper[5663]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:27:55.304504 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304387    5663 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:55.304504 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304503    5663 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:55.304504 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304510    5663 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304517    5663 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304521    5663 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304526    5663 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304540    5663 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304544    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304547    5663 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304550    5663 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304553    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304557    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304569    5663 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304575    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304579    5663 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304584    5663 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304588    5663 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304593    5663 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304598    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304603    5663 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304608    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:55.304690 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304612    5663 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304620    5663 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304626    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304630    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304635    5663 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304640    5663 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304645    5663 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304650    5663 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304655    5663 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304659    5663 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304664    5663 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304668    5663 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304673    5663 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304677    5663 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304681    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304685    5663 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304689    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304694    5663 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304699    5663 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:55.305553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304704    5663 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304708    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304714    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304719    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304722    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304731    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304734    5663 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304738    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304743    5663 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304747    5663 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304752    5663 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304756    5663 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304760    5663 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304765    5663 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304770    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304775    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304779    5663 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304783    5663 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304786    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304789    5663 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:55.306207 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304792    5663 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304796    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304801    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304805    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304809    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304814    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304818    5663 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304823    5663 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304827    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304832    5663 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304836    5663 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304840    5663 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304844    5663 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304848    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304852    5663 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304856    5663 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304860    5663 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304865    5663 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304875    5663 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304878    5663 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:55.307002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304881    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304883    5663 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304886    5663 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304889    5663 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304891    5663 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304894    5663 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304897    5663 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304899    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304902    5663 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304906    5663 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304910    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304915    5663 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304919    5663 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304923    5663 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304928    5663 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304932    5663 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304937    5663 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304942    5663 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304945    5663 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304948    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:55.307851 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304951    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304955    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304960    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304965    5663 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304969    5663 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.304978    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306396    5663 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306409    5663 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306414    5663 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306419    5663 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306424    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306429    5663 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306433    5663 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306437    5663 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306442    5663 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306447    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306451    5663 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306455    5663 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306459    5663 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306463    5663 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:55.308542 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306467    5663 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306472    5663 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306476    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306481    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306485    5663 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306489    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306493    5663 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306497    5663 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306501    5663 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306506    5663 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306510    5663 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306514    5663 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306518    5663 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306522    5663 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306526    5663 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306530    5663 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306534    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306538    5663 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306542    5663 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306546    5663 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:55.309418 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306550    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306554    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306559    5663 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306563    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306568    5663 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306573    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306578    5663 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306582    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306587    5663 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306591    5663 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306595    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306599    5663 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306603    5663 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306607    5663 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306610    5663 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306618    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306622    5663 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306626    5663 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306630    5663 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:55.310963 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306634    5663 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306640    5663 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306646    5663 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306651    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306656    5663 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306660    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306665    5663 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306669    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306674    5663 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306679    5663 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306683    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306687    5663 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306691    5663 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306696    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306700    5663 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306704    5663 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306712    5663 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306718    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:55.311782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306725    5663 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306730    5663 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306736    5663 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306741    5663 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306746    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306752    5663 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306756    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306761    5663 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306766    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306772    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306778    5663 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306782    5663 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306786    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306791    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306795    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306799    5663 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306803    5663 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306808    5663 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306812    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306817    5663 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:55.312798 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306821    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306826    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306830    5663 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306834    5663 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306838    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306843    5663 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306847    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306852    5663 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306857    5663 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306861    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306866    5663 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306870    5663 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306874    5663 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306881    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.306886    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307675    5663 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307686    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307690    5663 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307695    5663 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307700    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:55.313579 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307704    5663 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307708    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307717    5663 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307722    5663 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307727    5663 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307731    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307735    5663 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307740    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307744    5663 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307749    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307753    5663 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307757    5663 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307761    5663 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307768    5663 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307776    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307781    5663 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307785    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307790    5663 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307794    5663 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:55.314507 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307798    5663 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307803    5663 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307807    5663 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307812    5663 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307816    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307820    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307826    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307830    5663 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307834    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307838    5663 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307842    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307847    5663 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307850    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307854    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307859    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307865    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307869    5663 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307875    5663 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307879    5663 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:55.315339 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307884    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307888    5663 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307893    5663 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307897    5663 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307902    5663 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307906    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307910    5663 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307914    5663 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307918    5663 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307922    5663 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307927    5663 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307931    5663 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307935    5663 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307940    5663 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307945    5663 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307951    5663 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307957    5663 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307961    5663 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:55.316163 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307965    5663 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307969    5663 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307975    5663 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307979    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307982    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307988    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307992    5663 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.307997    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308000    5663 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308005    5663 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308010    5663 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308014    5663 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308018    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308022    5663 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308026    5663 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308030    5663 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308034    5663 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308038    5663 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308042    5663 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308046    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:55.316942 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308051    5663 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308055    5663 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308059    5663 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308063    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308067    5663 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308071    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308075    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308079    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308083    5663 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308087    5663 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308090    5663 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308094    5663 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308098    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308104    5663 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308108    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308113    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308117    5663 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308121    5663 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308133    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308139    5663 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:55.317830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308143    5663 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308147    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308151    5663 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308155    5663 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.308160    5663 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308301    5663 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308313    5663 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308323    5663 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308330    5663 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308337    5663 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308342    5663 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308366    5663 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308373    5663 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308378    5663 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308383    5663 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308388    5663 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308394    5663 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308399    5663 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308404    5663 flags.go:64] FLAG: --cgroup-root=""
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308408    5663 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308413    5663 flags.go:64] FLAG: --client-ca-file=""
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308417    5663 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308422    5663 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308428    5663 flags.go:64] FLAG: --cluster-domain=""
Oct 31 12:27:55.318702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308432    5663 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308440    5663 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308444    5663 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308449    5663 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308462    5663 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308467    5663 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308472    5663 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308477    5663 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308481    5663 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308487    5663 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308492    5663 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308497    5663 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308502    5663 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308509    5663 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308514    5663 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308519    5663 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308523    5663 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308528    5663 flags.go:64] FLAG: --enable-server="true"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308532    5663 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308541    5663 flags.go:64] FLAG: --event-burst="100"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308546    5663 flags.go:64] FLAG: --event-qps="50"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308551    5663 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308557    5663 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308563    5663 flags.go:64] FLAG: --eviction-hard=""
Oct 31 12:27:55.319731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308571    5663 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308577    5663 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308582    5663 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308587    5663 flags.go:64] FLAG: --eviction-soft=""
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308592    5663 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308597    5663 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308602    5663 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308607    5663 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308611    5663 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308617    5663 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308622    5663 flags.go:64] FLAG: --feature-gates=""
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308629    5663 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308635    5663 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308640    5663 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308645    5663 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308651    5663 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308657    5663 flags.go:64] FLAG: --help="false"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308664    5663 flags.go:64] FLAG: --hostname-override="ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308670    5663 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308675    5663 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308679    5663 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308685    5663 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308690    5663 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 12:27:55.320834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308695    5663 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308700    5663 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308704    5663 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308709    5663 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308715    5663 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308721    5663 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308725    5663 flags.go:64] FLAG: --kube-reserved=""
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308730    5663 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308735    5663 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308740    5663 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308745    5663 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308749    5663 flags.go:64] FLAG: --lock-file=""
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308754    5663 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308759    5663 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308763    5663 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308772    5663 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308777    5663 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308782    5663 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308788    5663 flags.go:64] FLAG: --logging-format="text"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308793    5663 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308798    5663 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308803    5663 flags.go:64] FLAG: --manifest-url=""
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308807    5663 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308815    5663 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308820    5663 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 12:27:55.321793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308826    5663 flags.go:64] FLAG: --max-pods="110"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308833    5663 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308837    5663 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308842    5663 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308846    5663 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308851    5663 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308856    5663 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308860    5663 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308871    5663 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308876    5663 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308881    5663 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308886    5663 flags.go:64] FLAG: --pod-cidr=""
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308891    5663 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308895    5663 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308900    5663 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308905    5663 flags.go:64] FLAG: --port="10250"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308909    5663 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308914    5663 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0504251c858d931d3"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308919    5663 flags.go:64] FLAG: --qos-reserved=""
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308924    5663 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308928    5663 flags.go:64] FLAG: --register-node="true"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308933    5663 flags.go:64] FLAG: --register-with-taints=""
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308939    5663 flags.go:64] FLAG: --registry-burst="10"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308943    5663 flags.go:64] FLAG: --registry-qps="5"
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308947    5663 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308952    5663 flags.go:64] FLAG: --reserved-memory=""
Oct 31 12:27:55.322813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308958    5663 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308963    5663 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308967    5663 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308972    5663 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308977    5663 flags.go:64] FLAG: --runonce="false"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308982    5663 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308987    5663 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308994    5663 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.308999    5663 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309004    5663 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309009    5663 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309014    5663 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309019    5663 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309024    5663 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309028    5663 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309033    5663 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309038    5663 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309043    5663 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309047    5663 flags.go:64] FLAG: --system-cgroups=""
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309052    5663 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309060    5663 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309065    5663 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309070    5663 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309077    5663 flags.go:64] FLAG: --tls-min-version=""
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309082    5663 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 12:27:55.323849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309086    5663 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309091    5663 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309095    5663 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309100    5663 flags.go:64] FLAG: --v="2"
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309106    5663 flags.go:64] FLAG: --version="false"
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309113    5663 flags.go:64] FLAG: --vmodule=""
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309119    5663 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309125    5663 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309330    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309338    5663 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309344    5663 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309369    5663 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309374    5663 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309379    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309387    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309392    5663 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309397    5663 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309401    5663 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309406    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309410    5663 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309416    5663 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:55.324898 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309421    5663 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309425    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309430    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309435    5663 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309440    5663 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309445    5663 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309450    5663 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309455    5663 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309459    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309463    5663 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309468    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309472    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309477    5663 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309481    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309486    5663 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309491    5663 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309496    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309500    5663 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309505    5663 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:55.325782 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309510    5663 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309515    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309519    5663 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309523    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309527    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309532    5663 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309539    5663 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309543    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309548    5663 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309552    5663 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309556    5663 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309561    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309567    5663 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309572    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309576    5663 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309580    5663 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309587    5663 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309593    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309598    5663 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309603    5663 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:55.326653 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309607    5663 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309611    5663 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309616    5663 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309620    5663 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309625    5663 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309631    5663 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309637    5663 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309642    5663 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309646    5663 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309650    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309654    5663 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309658    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309662    5663 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309667    5663 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309671    5663 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309675    5663 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309679    5663 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309683    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309689    5663 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309694    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:55.327494 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309699    5663 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309703    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309707    5663 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309711    5663 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309717    5663 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309722    5663 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309726    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309731    5663 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309735    5663 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309739    5663 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309744    5663 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309748    5663 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309752    5663 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309757    5663 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309762    5663 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309766    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309770    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309774    5663 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309779    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309783    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:55.328238 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309787    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309792    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309796    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309800    5663 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309804    5663 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309809    5663 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309813    5663 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309817    5663 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309821    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309825    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309830    5663 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309835    5663 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309840    5663 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309845    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.309892    5663 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309958    5663 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:55.329046 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309968    5663 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309973    5663 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309978    5663 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309983    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309988    5663 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309992    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.309996    5663 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310001    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310004    5663 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310008    5663 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310013    5663 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310017    5663 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310022    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310026    5663 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310029    5663 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310033    5663 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310037    5663 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310042    5663 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310046    5663 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310050    5663 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:55.329687 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310055    5663 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310059    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310063    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310067    5663 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310071    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310075    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310079    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310085    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310089    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310095    5663 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310099    5663 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310104    5663 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310110    5663 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310114    5663 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310118    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310123    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310127    5663 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310132    5663 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310136    5663 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310140    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:55.330520 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310144    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310149    5663 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310153    5663 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310157    5663 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310161    5663 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310165    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310169    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310174    5663 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310179    5663 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310183    5663 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310188    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310192    5663 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310196    5663 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310200    5663 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310205    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310210    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310215    5663 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310220    5663 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:55.331415 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310225    5663 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310231    5663 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310236    5663 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310240    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310245    5663 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310249    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310255    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310260    5663 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310265    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310269    5663 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310274    5663 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310280    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310284    5663 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310288    5663 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310293    5663 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310298    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310303    5663 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310307    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310312    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:55.332159 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310316    5663 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310321    5663 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310325    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310329    5663 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310333    5663 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310337    5663 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310342    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310347    5663 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310373    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310378    5663 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310382    5663 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310389    5663 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310396    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310401    5663 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310408    5663 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310412    5663 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310416    5663 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310420    5663 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310428    5663 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:55.332977 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310435    5663 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310441    5663 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310449    5663 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310454    5663 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310459    5663 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310463    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310467    5663 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310471    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.310475    5663 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.315780    5663 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.315794    5663 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.315817    5663 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.315836    5663 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315899    5663 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315904    5663 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315908    5663 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315911    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315914    5663 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315916    5663 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:55.333749 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315919    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315921    5663 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315924    5663 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315928    5663 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315933    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315935    5663 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315938    5663 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315941    5663 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315944    5663 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315947    5663 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315951    5663 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315956    5663 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315964    5663 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315970    5663 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315975    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315979    5663 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315982    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315985    5663 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315988    5663 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:55.334578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315990    5663 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315993    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315996    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.315998    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316001    5663 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316003    5663 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316006    5663 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316009    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316012    5663 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316014    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316017    5663 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316020    5663 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316022    5663 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316025    5663 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316027    5663 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316030    5663 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316033    5663 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316035    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316038    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316040    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:55.335324 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316043    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316045    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316048    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316051    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316053    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316056    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316060    5663 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316063    5663 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316065    5663 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316068    5663 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316071    5663 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316073    5663 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316076    5663 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316079    5663 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316083    5663 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316086    5663 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316090    5663 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316094    5663 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316099    5663 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316103    5663 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:55.336346 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316107    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316111    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316115    5663 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316119    5663 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316123    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316134    5663 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316139    5663 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316142    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316144    5663 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316147    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316149    5663 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316152    5663 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316155    5663 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316158    5663 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316160    5663 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316163    5663 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316166    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316169    5663 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316172    5663 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:55.337139 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316176    5663 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316179    5663 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316182    5663 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316184    5663 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316187    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316189    5663 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316193    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316195    5663 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316198    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316201    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316203    5663 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316208    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316212    5663 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316217    5663 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316221    5663 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316226    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316230    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316232    5663 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316235    5663 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:55.338035 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316238    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316242    5663 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316245    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.316276    5663 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316484    5663 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316494    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316501    5663 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316508    5663 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316515    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316520    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316525    5663 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316531    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316535    5663 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316539    5663 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316544    5663 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:55.338864 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316549    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316553    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316558    5663 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316562    5663 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316566    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316571    5663 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316575    5663 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316578    5663 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316581    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316583    5663 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316587    5663 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316591    5663 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316596    5663 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316600    5663 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316604    5663 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316608    5663 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316612    5663 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316619    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316624    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316628    5663 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:55.339450 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316632    5663 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316637    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316641    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316646    5663 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316650    5663 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316654    5663 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316659    5663 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316663    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316668    5663 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316674    5663 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316679    5663 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316683    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316688    5663 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316692    5663 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316696    5663 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316700    5663 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316705    5663 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316709    5663 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316713    5663 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316718    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:55.340168 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316722    5663 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316727    5663 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316734    5663 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316739    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316744    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316748    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316752    5663 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316757    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316762    5663 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316769    5663 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316773    5663 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316778    5663 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316783    5663 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316787    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316792    5663 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316797    5663 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316801    5663 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316806    5663 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316811    5663 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316816    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:55.341066 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316821    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316829    5663 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316834    5663 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316839    5663 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316844    5663 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316848    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316852    5663 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316854    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316857    5663 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316860    5663 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316863    5663 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316865    5663 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316868    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316872    5663 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316877    5663 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316882    5663 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316887    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316892    5663 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316897    5663 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316902    5663 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:55.341714 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316907    5663 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316912    5663 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316916    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316921    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316925    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316929    5663 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316933    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316937    5663 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316941    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316945    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316950    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316955    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.316959    5663 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317314    5663 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317320    5663 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.317389    5663 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:27:55.342486 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317442    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317499    5663 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317506    5663 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317512    5663 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317517    5663 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317523    5663 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317527    5663 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317532    5663 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317536    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317541    5663 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317547    5663 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317553    5663 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317559    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317564    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317567    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317570    5663 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317573    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317577    5663 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317587    5663 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317592    5663 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:27:55.343110 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317597    5663 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317601    5663 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317606    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317610    5663 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317615    5663 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317620    5663 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317625    5663 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317630    5663 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317634    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317638    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317642    5663 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317648    5663 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317654    5663 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317658    5663 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317663    5663 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317670    5663 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317675    5663 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317680    5663 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317686    5663 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:27:55.343675 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317690    5663 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317695    5663 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317698    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317701    5663 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317704    5663 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317708    5663 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317712    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317717    5663 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317722    5663 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317726    5663 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317730    5663 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317736    5663 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317740    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317745    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317749    5663 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317753    5663 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317757    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317760    5663 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317762    5663 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317765    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:27:55.344287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317767    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317770    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317773    5663 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317776    5663 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317782    5663 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317788    5663 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317793    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317797    5663 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317803    5663 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317813    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317818    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317824    5663 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317829    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317834    5663 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317839    5663 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317844    5663 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317847    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317850    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317852    5663 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:27:55.344984 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317855    5663 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317859    5663 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317862    5663 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317865    5663 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317869    5663 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317872    5663 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317876    5663 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317878    5663 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317881    5663 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317883    5663 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317886    5663 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317889    5663 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317891    5663 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317894    5663 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317897    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317900    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317904    5663 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317910    5663 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317915    5663 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317919    5663 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:27:55.345662 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317924    5663 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317929    5663 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317934    5663 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317939    5663 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317944    5663 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317949    5663 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317956    5663 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:27:55.317961    5663 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.318035    5663 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.319378    5663 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.319457    5663 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.319711    5663 server.go:1014] "Starting client certificate rotation"
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.319872    5663 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.320026    5663 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:21:40 +0000 UTC" deadline="2028-06-12 12:53:30.496518041 +0000 UTC"
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.320107    5663 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14160h25m35.176413643s"
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.320422    5663 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 12:27:55.346558 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.322071    5663 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 12:27:55.347284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.327625    5663 log.go:25] "Validated CRI v1 runtime API"
Oct 31 12:27:55.347284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.329850    5663 log.go:25] "Validated CRI v1 image API"
Oct 31 12:27:55.347284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.330866    5663 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 12:27:55.347284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.333456    5663 fs.go:135] Filesystem UUIDs: map[4e5a7c99-e7fa-4f42-8971-e54d56bd69d6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 83d322fb-9b6a-4354-ae1d-eb03037c0058:/dev/nvme0n1p3]
Oct 31 12:27:55.347493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.333482    5663 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01a891d83b9d1e5beb4d2163fdec8add534ee2db0c2ffa135000fab8ab56a60a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01a891d83b9d1e5beb4d2163fdec8add534ee2db0c2ffa135000fab8ab56a60a/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02ae7b8518a5413a0f029f81d85a276fc34e2bb806662d3112a4dc68973b1ea6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02ae7b8518a5413a0f029f81d85a276fc34e2bb806662d3112a4dc68973b1ea6/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08191ab633083238796092e5c3633512d3b6eb190d2f27ed6431ff7014628ede/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08191ab633083238796092e5c3633512d3b6eb190d2f27ed6431ff7014628ede/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08e7cb028de0cc208ae8b278f08edc9368fa95b759a296047045c847c4f0d33f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08e7cb028de0cc208ae8b278f08edc9368fa95b759a296047045c847c4f0d33f/userdata/shm major:0 minor:273 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1749c085df22cc8d63e2372be0379fde981c06b1ea93704510a25f6359689ee2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1749c085df22cc8d63e2372be0379fde981c06b1ea93704510a25f6359689ee2/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fc122e5f4ab142e986ebb5a8f5304e189d3616d70df6d9bcdd26cceab90e590/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fc122e5f4ab142e986ebb5a8f5304e189d3616d70df6d9bcdd26cceab90e590/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/388243f74d05102978315458c3adbd21e98488457e1843d8feceaa5405b877ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/388243f74d05102978315458c3adbd21e98488457e1843d8feceaa5405b877ce/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5dbf68e07df4d2fd91d150bbff7ff5cb9c41d281ddb536ab6c39c475e9af0408/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5dbf68e07df4d2fd91d150bbff7ff5cb9c41d281ddb536ab6c39c475e9af0408/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6de405225e6586537bf4bb3fd5fd4d77051b5ad74447bedbda16bdd625aebd65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6de405225e6586537bf4bb3fd5fd4d77051b5ad74447bedbda16bdd625aebd65/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ae1e5a6300d7f843f3e921d46b6523eecb9a454926f0d8d30153a72639ebd10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ae1e5a6300d7f843f3e921d46b6523eecb9a454926f0d8d30153a72639ebd10/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f4c70cafe06d740f8605ade053156e1482bb725fcc650779c2bb03b1914ab15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f4c70cafe06d740f8605ade053156e1482bb725fcc650779c2bb03b1914ab15/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8fd7254459e0d5033b1468aa33d32890f9898ca005969bb0e58c9b8cad878d50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8fd7254459e0d5033b1468aa33d32890f9898ca005969bb0e58c9b8cad878d50/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/903f93924da898fa053ab86d157eb948eb84188f5343dbfa4cfeed29fb7211e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/903f93924da898fa053ab86d157eb948eb84188f5343dbfa4cfeed29fb7211e5/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95ebfde83dc014dfcdd70f0a0603335cd77ac6fc0b1b1ca9342a3535e5722674/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95ebfde83dc014dfcdd70f0a0603335cd77ac6fc0b1b1ca9342a3535e5722674/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d400fd1ede94d33680989708eec336060c5b70f64cf2ddd8895a4bcf58b7b83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d400fd1ede94d33680989708eec336060c5b70f64cf2ddd8895a4bcf58b7b83/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a715c33150a3ce754318231e430b41fff8dc2efee8cd5a01206ad8f2ea204924/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a715c33150a3ce754318231e430b41fff8dc2efee8cd5a01206ad8f2ea204924/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab06aedaec96eebe2acde541f69eef57a4330288fe1b90f919e2f183a386e916/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab06aedaec96eebe2acde541f69eef57a4330288fe1b90f919e2f183a386e916/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b957d88582d3c25c206d8852f928e9f9ad8952db75bc492b45f7dc21f76aaf68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b957d88582d3c25c206d8852f928e9f9ad8952db75bc492b45f7dc21f76aaf68/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5b8b9b880947cf81957233c5d94ee1156d415144bf00c0eb1eb667c7949e657/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5b8b9b880947cf81957233c5d94ee1156d415144bf00c0eb1eb667c7949e657/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d096c0ea1eaf621fa65e102e73723edbc2373d4feb522719fb85712a6c3450c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d096c0ea1eaf621fa65e102e73723edbc2373d4feb522719fb85712a6c3450c7/userdata/shm major:0 minor:327 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d16185dc0451c4f52bf242e824c1beefc35ba756c3b32bfbd8a6e3a3906fd6eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d16185dc0451c4f52bf242e824c1beefc35ba756c3b32bfbd8a6e3a3906fd6eb/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f494df66607706fc97ff0268894b86d3efeac283af0fb58b605eb1ad9583f969/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f494df66607706fc97ff0268894b86d3efeac283af0fb58b605eb1ad9583f969/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb8772d40acb2f2af7233e333108df2a91d1cd8b56a70e62945993c76bd75a32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb8772d40acb2f2af7233e333108df2a91d1cd8b56a70e62945993c76bd75a32/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0589f9a7-c2b8-408f-8374-a25b489ba69a/volumes/kubernetes.io~projected/kube-api-access-8wvqq:{mountpoint:/var/lib/kubelet/pods/0589f9a7-c2b8-408f-8374-a25b489ba69a/volumes/kubernetes.io~projected/kube-api-access-8wvqq major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0589f9a7-c2b8-408f-8374-a25b489ba69a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0589f9a7-c2b8-408f-8374-a25b489ba69a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0589f9a7-c2b8-408f-8374-a25b489ba69a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/0589f9a7-c2b8-408f-8374-a25b489ba69a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b8fc463-aafe-4750-aaeb-827522735a5e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0b8fc463-aafe-4750-aaeb-827522735a5e/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/119a3a49-2f9f-4285-bba5-e035d61b03cd/volumes/kubernetes.io~projected/kube-api-access-gljhb:{mountpoint:/var/lib/kubelet/pods/119a3a49-2f9f-4285-bba5-e035d61b03cd/volumes/kubernetes.io~projected/kube-api-access-gljhb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1507e7d6-a6e4-4cb9-adf5-c24d8619cede/volumes/kubernetes.io~projected/kube-api-access-gmtfl:{mountpoint:/var/lib/kubelet/pods/1507e7d6-a6e4-4cb9-adf5-c24d8619cede/volumes/kubernetes.io~projected/kube-api-access-gmtfl major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1507e7d6-a6e4-4cb9-adf5-c24d8619cede/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/1507e7d6-a6e4-4cb9-adf5-c24d8619cede/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1507e7d6-a6e4-4cb9-adf5-c24d8619cede/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/1507e7d6-a6e4-4cb9-adf5-c24d8619cede/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1507e7d6-a6e4-4cb9-adf5-c24d8619cede/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/1507e7d6-a6e4-4cb9-adf5-c24d8619cede/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23cc7cc6-1673-434b-a56f-e83298dc5d91/volumes/kubernetes.io~projected/kube-api-access-l822j:{mountpoint:/var/lib/kubelet/pods/23cc7cc6-1673-434b-a56f-e83298dc5d91/volumes/kubernetes.io~projected/kube-api-access-l822j major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4be38679-7629-4ec8-aed1-d2ef3fcd8a10/volumes/kubernetes.io~projected/kube-api-access-cxv7f:{mountpoint:/var/lib/kubelet/pods/4be38679-7629-4ec8-aed1-d2ef3fcd8a10/volumes/kubernetes.io~projected/kube-api-access-cxv7f major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/542d428b-b94d-4033-9805-d13da84ede85/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/542d428b-b94d-4033-9805-d13da84ede85/volumes/kubernetes.io~secret/tls-certificates major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59c1bd59-6c93-426a-9e81-a410a9cc62f4/volumes/kubernetes.io~projected/kube-api-access-fhf7n:{mountpoint:/var/lib/kubelet/pods/59c1bd59-6c93-426a-9e81-a410a9cc62f4/volumes/kubernetes.io~projected/kube-api-access-fhf7n major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59c1bd59-6c93-426a-9e81-a410a9cc62f4/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/59c1bd59-6c93-426a-9e81-a410a9cc62f4/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59c1bd59-6c93-426a-9e81-a410a9cc62f4/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/59c1bd59-6c93-426a-9e81-a410a9cc62f4/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a733526-fcbe-483b-8e2a-e58c99be5707/volumes/kubernetes.io~projected/kube-api-access-h8gb9:{mountpoint:/var/lib/kubelet/pods/5a733526-fcbe-483b-8e2a-e58c99be5707/volumes/kubernetes.io~projected/kube-api-access-h8gb9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026/volumes/kubernetes.io~projected/kube-api-access-nzvjn:{mountpoint:/var/lib/kubelet/pods/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026/volumes/kubernetes.io~projected/kube-api-access-nzvjn major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/609830fb-fa9a-4886-89be-cb40635b2c67/volumes/kubernetes.io~projected/kube-api-access-2shhr:{mountpoint:/var/lib/kubelet/pods/609830fb-fa9a-4886-89be-cb40635b2c67/volumes/kubernetes.io~projected/kube-api-access-2shhr major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/609830fb-fa9a-4886-89be-cb40635b2c67/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/609830fb-fa9a-4886-89be-cb40635b2c67/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7287c3ee-7c75-47e2-9c34-d436193f32e3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7287c3ee-7c75-47e2-9c34-d436193f32e3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7287c3ee-7c75-47e2-9c34-d436193f32e3/volumes/kubernetes.io~projected/kube-api-access-c4n6f:{mountpoint:/var/lib/kubelet/pods/7287c3ee-7c75-47e2-9c34-d436193f32e3/volumes/kubernetes.io~projected/kube-api-access-c4n6f major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7287c3ee-7c75-47e2-9c34-d436193f32e3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7287c3ee-7c75-47e2-9c34-d436193f32e3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c53a9b9-23b0-4598-8ee6-35de80dd12d4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8c53a9b9-23b0-4598-8ee6-35de80dd12d4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95feab9f-adf7-4d34-bde6-d0a1b450df6c/volumes/kubernetes.io~projected/kube-api-access-kvxxz:{mountpoint:/var/lib/kubelet/pods/95feab9f-adf7-4d34-bde6-d0a1b450df6c/volumes/kubernetes.io~projected/kube-api-access-kvxxz major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95feab9f-adf7-4d34-bde6-d0a1b450df6c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/95feab9f-adf7-4d34-bde6-d0a1b450df6c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3ba234a-bb37-4ed4-a505-4a8494d4e862/volumes/kubernetes.io~projected/kube-api-access-l5tbp:{mountpoint:/var/lib/kubelet/pods/a3ba234a-bb37-4ed4-a505-4a8494d4e862/volumes/kubernetes.io~projected/kube-api-access-l5tbp major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3ba234a-bb37-4ed4-a505-4a8494d4e862/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a3ba234a-bb37-4ed4-a505-4a8494d4e862/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3ba234a-bb37-4ed4-a505-4a8494d4e862/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/a3ba234a-bb37-4ed4-a505-4a8494d4e862/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab9d44df-91e2-4e21-afd7-eac98bf106e5/volumes/kubernetes.io~projected/kube-api-access-httpx:{mountpoint:/var/lib/kubelet/pods/ab9d44df-91e2-4e21-afd7-eac98bf106e5/volumes/kubernetes.io~projected/kube-api-access-httpx major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adb879bf-aeb5-468c-8f93-4cb40d6e4e36/volumes/kubernetes.io~projected/kube-api-access-2vpf2:{mountpoint:/var/lib/kubelet/pods/adb879bf-aeb5-468c-8f93-4cb40d6e4e36/volumes/kubernetes.io~projected/kube-api-access-2vpf2 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adb879bf-aeb5-468c-8f93-4cb40d6e4e36/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/adb879bf-aeb5-468c-8f93-4cb40d6e4e36/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a/volumes/kubernetes.io~projected/kube-api-access-4jk2d:{mountpoint:/var/lib/kubelet/pods/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a/volumes/kubernetes.io~projected/kube-api-access-4jk2d major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~projected/kube-api-access-zsbvh:{mountpoint:/var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~projected/kube-api-access-zsbvh major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~projected/registry-tls major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9/volumes/kubernetes.io~projected/kube-api-access-kcjhz:{mountpoint:/var/lib/kubelet/pods/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9/volumes/kubernetes.io~projected/kube-api-access-kcjhz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5bb3612-89f9-4494-9634-2b8bed367954/volumes/kubernetes.io~projected/kube-api-access-c4j49:{mountpoint:/var/lib/kubelet/pods/f5bb3612-89f9-4494-9634-2b8bed367954/volumes/kubernetes.io~projected/kube-api-access-c4j49 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5bb3612-89f9-4494-9634-2b8bed367954/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/f5bb3612-89f9-4494-9634-2b8bed367954/volumes/kubernetes.io~secret/metrics-tls major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe072ecb-08a7-4c60-ad8b-d6643882ba73/volumes/kubernetes.io~projected/kube-api-access-m4958:{mountpoint:/var/lib/kubelet/pods/fe072ecb-08a7-4c60-ad8b-d6643882ba73/volumes/kubernetes.io~projected/kube-api-access-m4958 major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a2528bdeafa07aad3acfb3b2c6b2ecff71665f402ebf44882a2006bcbe45fcba/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/65fa40512e734d2209a59bf906942b4ac5b424788d7c34472c02f8fa695b4052/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/fbba956e99a64846c9f4a55f7898ac6d6f85b9ec77c957048a9f8622c465436e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/d7ebc5d7a67893a532457ac9bf52dec97e0792ee504537f799b93f89560d0678/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/816374574a7eefb86e437785b176b54aae58d8842514a9633c15b56269d5b4fb/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3f7f19e1deca820b2ef554081e3e7b747c092393b1b29b154b526448175134d1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bc6289484956b1287b2f5e86fabff7659a0262fd891526dc564d346badb52c18/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/78a5f637235a946a4be3e45c6c732853069c0e78d5fa83e89ac056b2cc6cd966/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2fd1d8303ecb8ac737d21205b80a7ed205cfd9398ad491a7737d46e6407ccbd1/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/a037a93819fddc139ce4a20330c0f47bac9a0f0c200ae3143cea3d49ee5f51db/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/051b98435c1205bc400eea6a1bffd9319619d4dff1f6a983a5abf35ccd5eaa3c/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/f3d7b7d69a034ed646fce2d55e6dafc8311b9d91e9d78faebc91f5c483206cc7/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/919f9ebbf26f9818a0af75f4a637d4659d47eed6da20707830f74529a3f35f9b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c009b786de5864b750560fa233ad60fe0c5d0584b5d070c70a85094e5a7a3d54/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6568d794d4652bdba374dcc9325c5249ec8a3da6990d54061cbe6ef3396e98de/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/65b6be12ba32b2d0a3fa8108c12c3ccc4193fc215b4e8e8f10117913ac15d998/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7d7b605bf8043f4932a7304ed5cf032a9917aad5ccc8160319e64b7a797310a3/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f0fbcaba614fa1bfb10c5c91da96bb4918f23f56b5670aa8205e1bd1ae29ec8a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8cbb9a7f30ead0acfd0621e4b09b004fdf10b77d5e5b025f2239904069fc63dd/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/54336cfa43d7ca1d834ac8eb4802a66f8d2a7fe6105af07fb2c7fba698bf067e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b092b6e0628906d66cf4feb82b2baaae352d700cd2fd5b9225e86af2108979f7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/e4aad5d25204c3270709033c0f49b863a7f38139cfadfabba4f225cc45338afe/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/566a64a6998668dc9e3a70fe44f772d05eaaf0de1fd135cbde076fc3413de06a/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/fcbc4a51f3e4cac6211dbbecce55867724e5663ca1e56f3bcb0821774ad74b93/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/788096cd1c1969ee884d276c9100a4f323cde6c032942dcb1d755561106a3ab3/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/79ceaee945a0be63573924af5fc0bd0c775811da076c3efc2a89cc9e2cf37602/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/08e82bbd79fefbd97a310ea849751b55f54c744ba483f548d0d28b61decf1ac5/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/aa399df5dcb525fb7d535e4b19763553b08c10789c6135f91bee833f236bdde8/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/84c0843266636cf0f1a9032ce721975f60cdfc0205d9995c70115d25b00f14b1/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/60a8a39da886075be36923c56e09bd8f2a8543551f6ce2ec24f41784a18161fa/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/d768636dfbf5f74ba19d213ea1c021ebcf33bb8b730336e1764f03c9f4357044/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/4d9966e64579ac171399dad896e32bfb5f33a74723f0f758bd13e5b1b4cf428b/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/ec6fe071dd887607af1d21d32fa1aa72afbb9eee379c10af67fcc50f9d88a1ba/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/60b322197bc514251b347c78827f506ea3131f9f5dba35e15095b9b3f1c8e340/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/c8d2ea3cf3832ae30aa9d4a7eb727e60ff2fc9c5836692ee9bb3252c347b1074/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/9a0f29cdcb8c28498d882a23f515a114aa50a69b3a8b0c26fc4723d5bba59f04/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/add63e2d39a50999da17e0c4b7e6930f883282b92db3800bed5003e5b0a7dd0e/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/871c238ad998ad1b551ebd6af07a4059d134c6cb83d156adf8184c6b9b8ca015/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/274bffc0fe855d11c28d8f3e6609acdeab6038b1f3bf1c686e27ca6ad33a340e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/c150dd6118ca4fe73b7f3ce148757dae23885976dcfd5539ccdefbf4511349c4/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/75dbf065e646da08ce6e59772ac1b085cb4e3556ca354f34406ffd324af91a0e/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/ea55a4872e15499a950c41a06c6a01140bb584118a4f5918c2af038d911dc195/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/2ee6928cf1cb6eca3390ce0349a0f0f2fbf7f394ddbf65540aec0c19e0d51312/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/a5d5a9d2a26f279a8c363f0bc93907500afae5907f59be14aac98f5853ff2048/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/1ea77783558bd3f0d0022542d7864294cd5cdce6c24651304d427169efb39fe9/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/572bb620a0f9cb37b5cb59d327f93360d7f143160f089f491e198db303dcb61e/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/3cbb24d395182fdbfa59c62af7e090fb5c63bb640428f4eb23261f5937025941/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/9cf74a3e346cc770e12e8f1c0e0d184b01546e6212fe7bcdb1486d42d192f131/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/0508aab9199a6261042a47afaf0fa53f42ca3b93506eb760d2c9d504be6446c5/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/2666575c0dc1ad2f625b70353784a4c3b9aabc7f79f1b61c778abb928b84d5cc/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/fe470906412648051669835467dc8e8f29f8c3938059b4f88925f1590a67ce29/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/7722cc365f7ca9cb1858b0c8736d3303be57e026545653a2d3899b7d52df5d56/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ab377d86aa251b7a34ee7a4988b1a127ace857a708ed8e457ca05db44bf7bc80/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/dcc31c19195956ebdd57a29a61ba21a1c3fa6cac4dd82e5776a2390454f68165/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/09b208e466e4088ff00861eaa213ad8131e0ccfec73b19cb4c415ba7a21d2e00/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/ce53d9d273e3870bd068f8c278c5210daea3b3fd2d305da074ed9818ac45d070/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/37cc9bef478135ca965efa9ade058730abc0f5d36cebe2967457d317a897f5a3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c2aa14fd364f97ee185c1294516d4760a85f5698b245d39b9a0b76e08b5b2225/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/317af8e2ab7357f6d5371d3403a914da3c76837fda885f9fcac8ec1e2d4aca83/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/4351e6e7579967600cde2bcc0b01afba34f68f2c920e3da88f77327ad2d59af8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/38f4865d9cb7e7eb5917dbe27bd3208247d7d115940b13c4cc0a92597460a136/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.343792    5663 manager.go:217] Machine: {Timestamp:2026-10-31 12:27:55.343096827 +0000 UTC m=+0.083523041 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100206 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f70d8964258dbd79cc017389f7f9 SystemUUID:ec24f70d-8964-258d-bd79-cc017389f7f9 BootID:e56e40ab-3640-4186-b1ed-abcb38693e80 Filesystems:[{Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7287c3ee-7c75-47e2-9c34-d436193f32e3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95feab9f-adf7-4d34-bde6-d0a1b450df6c/volumes/kubernetes.io~projected/kube-api-access-kvxxz DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:239 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1507e7d6-a6e4-4cb9-adf5-c24d8619cede/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1507e7d6-a6e4-4cb9-adf5-c24d8619cede/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/23cc7cc6-1673-434b-a56f-e83298dc5d91/volumes/kubernetes.io~projected/kube-api-access-l822j DeviceMajor:0 DeviceMinor:326 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adb879bf-aeb5-468c-8f93-4cb40d6e4e36/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/388243f74d05102978315458c3adbd21e98488457e1843d8feceaa5405b877ce/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/609830fb-fa9a-4886-89be-cb40635b2c67/volumes/kubernetes.io~projected/kube-api-access-2shhr DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6de405225e6586537bf4bb3fd5fd4d77051b5ad74447bedbda16bdd625aebd65/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f4c70cafe06d740f8605ade053156e1482bb725fcc650779c2bb03b1914ab15/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f494df66607706fc97ff0268894b86d3efeac283af0fb58b605eb1ad9583f969/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0589f9a7-c2b8-408f-8374-a25b489ba69a/volumes/kubernetes.io~projected/kube-api-access-8wvqq DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d400fd1ede94d33680989708eec336060c5b70f64cf2ddd8895a4bcf58b7b83/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f5bb3612-89f9-4494-9634-2b8bed367954/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb8772d40acb2f2af7233e333108df2a91d1cd8b56a70e62945993c76bd75a32/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0589f9a7-c2b8-408f-8374-a25b489ba69a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:301 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/59c1bd59-6c93-426a-9e81-a410a9cc62f4/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:353 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3ba234a-bb37-4ed4-a505-4a8494d4e862/volumes/kubernetes.io~projected/kube-api-access-l5tbp DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0589f9a7-c2b8-408f-8374-a25b489ba69a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:238 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02ae7b8518a5413a0f029f81d85a276fc34e2bb806662d3112a4dc68973b1ea6/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01a891d83b9d1e5beb4d2163fdec8add534ee2db0c2ffa135000fab8ab56a60a/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5dbf68e07df4d2fd91d150bbff7ff5cb9c41d281ddb536ab6c39c475e9af0408/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1507e7d6-a6e4-4cb9-adf5-c24d8619cede/volumes/kubernetes.io~projected/kube-api-access-gmtfl DeviceMajor:0 DeviceMinor:319 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fc122e5f4ab142e986ebb5a8f5304e189d3616d70df6d9bcdd26cceab90e590/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59c1bd59-6c93-426a-9e81-a410a9cc62f4/volumes/kubernetes.io~projected/kube-api-access-fhf7n DeviceMajor:0 DeviceMinor:358 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adb879bf-aeb5-468c-8f93-4cb40d6e4e36/volumes/kubernetes.io~projected/kube-api-access-2vpf2 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a715c33150a3ce754318231e430b41fff8dc2efee8cd5a01206ad8f2ea204924/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/542d428b-b94d-4033-9805-d13da84ede85/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3ba234a-bb37-4ed4-a505-4a8494d4e862/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8fd7254459e0d5033b1468aa33d32890f9898ca005969bb0e58c9b8cad878d50/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d16185dc0451c4f52bf242e824c1beefc35ba756c3b32bfbd8a6e3a3906fd6eb/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1749c085df22cc8d63e2372be0379fde981c06b1ea93704510a25f6359689ee2/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~projected/kube-api-access-zsbvh DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1507e7d6-a6e4-4cb9-adf5-c24d8619cede/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:318 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a733526-fcbe-483b-8e2a-e58c99be5707/volumes/kubernetes.io~projected/kube-api-access-h8gb9 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95ebfde83dc014dfcdd70f0a0603335cd77ac6fc0b1b1ca9342a3535e5722674/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ae1e5a6300d7f843f3e921d46b6523eecb9a454926f0d8d30153a72639ebd10/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9/volumes/kubernetes.io~projected/kube-api-access-kcjhz DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f5bb3612-89f9-4494-9634-2b8bed367954/volumes/kubernetes.io~projected/kube-api-access-c4j49 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab06aedaec96eebe2acde541f69eef57a4330288fe1b90f919e2f183a386e916/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5b8b9b880947cf81957233c5d94ee1156d415144bf00c0eb1eb667c7949e657/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7287c3ee-7c75-47e2-9c34-d436193f32e3/volumes/kubernetes.io~projected/kube-api-access-c4n6f DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95feab9f-adf7-4d34-bde6-d0a1b450df6c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be73966c-8b20-4130-8203-5cf6ca09df27/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b957d88582d3c25c206d8852f928e9f9ad8952db75bc492b45f7dc21f76aaf68/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d096c0ea1eaf621fa65e102e73723edbc2373d4feb522719fb85712a6c3450c7/userdata/shm DeviceMajor:0 DeviceMinor:327 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59c1bd59-6c93-426a-9e81-a410a9cc62f4/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:357 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0b8fc463-aafe-4750-aaeb-827522735a5e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a/volumes/kubernetes.io~projected/kube-api-access-4jk2d DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab9d44df-91e2-4e21-afd7-eac98bf106e5/volumes/kubernetes.io~projected/kube-api-access-httpx DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/119a3a49-2f9f-4285-bba5-e035d61b03cd/volumes/kubernetes.io~projected/kube-api-access-gljhb DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7287c3ee-7c75-47e2-9c34-d436193f32e3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/903f93924da898fa053ab86d157eb948eb84188f5343dbfa4cfeed29fb7211e5/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/609830fb-fa9a-4886-89be-cb40635b2c67/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026/volumes/kubernetes.io~projected/kube-api-access-nzvjn DeviceMajor:0 DeviceMinor:242 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a3ba234a-bb37-4ed4-a505-4a8494d4e862/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:266 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe072ecb-08a7-4c60-ad8b-d6643882ba73/volumes/kubernetes.io~projected/kube-api-access-m4958 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08e7cb028de0cc208ae8b278f08edc9368fa95b759a296047045c847c4f0d33f/userdata/shm DeviceMajor:0 DeviceMinor:273 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8c53a9b9-23b0-4598-8ee6-35de80dd12d4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:324 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4be38679-7629-4ec8-aed1-d2ef3fcd8a10/volumes/kubernetes.io~projected/kube-api-access-cxv7f DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08191ab633083238796092e5c3633512d3b6eb190d2f27ed6431ff7014628ede/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01a891d83b9d1e5 MacAddress:76:42:98:b7:ba:fb Speed:10000 Mtu:8901} {Name:08e7cb028de0cc2 MacAddress:6a:14:f2:1d:2a:03 Speed:10000 Mtu:8901} {Name:1749c085df22cc8 MacAddress:16:3c:de:11:5d:ea Speed:10000 Mtu:8901} {Name:2fc122e5f4ab142 MacAddress:96:c2:34:7b:26:34 Speed:10000 Mtu:8901} {Name:5dbf68e07df4d2f MacAddress:4a:11:59:05:7b:17 Speed:10000 Mtu:8901} {Name:95ebfde83dc014d MacAddress:4e:0a:05:5f:7c:49 Speed:10000 Mtu:8901} {Name:9d400fd1ede94d3 MacAddress:b6:53:25:a1:82:4c Speed:10000 Mtu:8901} {Name:ab06aedaec96eeb MacAddress:a6:0c:61:23:88:8d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d9:68:d1:43 Speed:0 Mtu:9001} {Name:br-int MacAddress:62:1c:78:df:18:71 Speed:0 Mtu:8901} {Name:c5b8b9b880947cf MacAddress:1e:18:d4:b2:e4:2b Speed:10000 Mtu:8901} {Name:d096c0ea1eaf621 MacAddress:de:98:26:9f:3c:c8 Speed:10000 Mtu:8901} {Name:d16185dc0451c4f MacAddress:c2:df:55:96:8e:33 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d9:68:d1:43 Speed:0 Mtu:9001} {Name:fb8772d40acb2f2 MacAddress:aa:e2:b4:7c:fe:34 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a2:0b:13:66:8a:6b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:0a:dc:4e:8e:30:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.344306    5663 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.344384    5663 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.344774    5663 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.344799    5663 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-137.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.345040    5663 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.345053    5663 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.345073    5663 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.345089    5663 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.345190    5663 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.345413    5663 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.345562    5663 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.345613    5663 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.345627    5663 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.345641    5663 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.345655    5663 kubelet.go:405] "Adding apiserver pod source"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.345667    5663 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.347614    5663 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 12:27:55.347819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.347627    5663 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 12:27:55.348961 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.348319    5663 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364080    5663 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364099    5663 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364241    5663 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364526    5663 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364571    5663 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364704    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364723    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364733    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364742    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364751    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364761    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364771    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364778    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364798    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364808    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364820    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364833    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364862    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 12:27:55.365381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.364872    5663 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 12:27:55.366936 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.366303    5663 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:27:55.368522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.368499    5663 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:27:55.369700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.369677    5663 server.go:1263] "Started kubelet"
Oct 31 12:27:55.369994 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.369970    5663 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 12:27:55.370127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.370078    5663 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 12:27:55.370239 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.370146    5663 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 12:27:55.370144 ip-10-0-140-137 systemd[1]: Started Kubernetes Kubelet.
Oct 31 12:27:55.370486 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.370469    5663 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 12:27:55.372453 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.372082    5663 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 12:27:55.376956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.373009    5663 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 12:27:55.376956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.373021    5663 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 12:27:55.376956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.373083    5663 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:21:42 +0000 UTC" deadline="2028-05-12 19:33:54.869739335 +0000 UTC"
Oct 31 12:27:55.376956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.373116    5663 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13423h5m59.496627259s"
Oct 31 12:27:55.376956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.373151    5663 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 12:27:55.376956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.373174    5663 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 12:27:55.376956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.373290    5663 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 12:27:55.376956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.375048    5663 factory.go:55] Registering systemd factory
Oct 31 12:27:55.376956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.375078    5663 factory.go:223] Registration of the systemd container factory successfully
Oct 31 12:27:55.377463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.377427    5663 factory.go:153] Registering CRI-O factory
Oct 31 12:27:55.377463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.377452    5663 factory.go:223] Registration of the crio container factory successfully
Oct 31 12:27:55.377560 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.377532    5663 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 12:27:55.377560 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.377558    5663 factory.go:103] Registering Raw factory
Oct 31 12:27:55.377618 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.377581    5663 manager.go:1199] Started watching for new ooms in manager
Oct 31 12:27:55.378024 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.377972    5663 manager.go:319] Starting recovery of all containers
Oct 31 12:27:55.387742 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.387505    5663 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:27:55.388739 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.388696    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="609830fb-fa9a-4886-89be-cb40635b2c67" volumeName="kubernetes.io/projected/609830fb-fa9a-4886-89be-cb40635b2c67-kube-api-access-2shhr" seLinuxMountContext=""
Oct 31 12:27:55.389016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.388744    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adb879bf-aeb5-468c-8f93-4cb40d6e4e36" volumeName="kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 12:27:55.389016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.388779    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfebfdf1-70e8-4540-ba01-67df6d7fd9a9" volumeName="kubernetes.io/empty-dir/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-tmp" seLinuxMountContext=""
Oct 31 12:27:55.389016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.388795    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe072ecb-08a7-4c60-ad8b-d6643882ba73" volumeName="kubernetes.io/projected/fe072ecb-08a7-4c60-ad8b-d6643882ba73-kube-api-access-m4958" seLinuxMountContext=""
Oct 31 12:27:55.389016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.388810    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1507e7d6-a6e4-4cb9-adf5-c24d8619cede" volumeName="kubernetes.io/configmap/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 12:27:55.389016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.388853    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59c1bd59-6c93-426a-9e81-a410a9cc62f4" volumeName="kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 12:27:55.389016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.388871    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="609830fb-fa9a-4886-89be-cb40635b2c67" volumeName="kubernetes.io/secret/609830fb-fa9a-4886-89be-cb40635b2c67-cert" seLinuxMountContext=""
Oct 31 12:27:55.389016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.388883    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be73966c-8b20-4130-8203-5cf6ca09df27" volumeName="kubernetes.io/secret/be73966c-8b20-4130-8203-5cf6ca09df27-installation-pull-secrets" seLinuxMountContext=""
Oct 31 12:27:55.389016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.388899    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfebfdf1-70e8-4540-ba01-67df6d7fd9a9" volumeName="kubernetes.io/empty-dir/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-tuned" seLinuxMountContext=""
Oct 31 12:27:55.389016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.388933    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5bb3612-89f9-4494-9634-2b8bed367954" volumeName="kubernetes.io/projected/f5bb3612-89f9-4494-9634-2b8bed367954-kube-api-access-c4j49" seLinuxMountContext=""
Oct 31 12:27:55.389016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.388953    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23cc7cc6-1673-434b-a56f-e83298dc5d91" volumeName="kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389021    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95feab9f-adf7-4d34-bde6-d0a1b450df6c" volumeName="kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389037    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adb879bf-aeb5-468c-8f93-4cb40d6e4e36" volumeName="kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389050    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0589f9a7-c2b8-408f-8374-a25b489ba69a" volumeName="kubernetes.io/empty-dir/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-textfile" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389067    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59c1bd59-6c93-426a-9e81-a410a9cc62f4" volumeName="kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-oauth-config" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389097    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3ba234a-bb37-4ed4-a505-4a8494d4e862" volumeName="kubernetes.io/configmap/a3ba234a-bb37-4ed4-a505-4a8494d4e862-metrics-client-ca" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389114    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab9d44df-91e2-4e21-afd7-eac98bf106e5" volumeName="kubernetes.io/empty-dir/ab9d44df-91e2-4e21-afd7-eac98bf106e5-tmp-dir" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389126    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0589f9a7-c2b8-408f-8374-a25b489ba69a" volumeName="kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-tls" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389140    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="119a3a49-2f9f-4285-bba5-e035d61b03cd" volumeName="kubernetes.io/projected/119a3a49-2f9f-4285-bba5-e035d61b03cd-kube-api-access-gljhb" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389153    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="542d428b-b94d-4033-9805-d13da84ede85" volumeName="kubernetes.io/secret/542d428b-b94d-4033-9805-d13da84ede85-tls-certificates" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389191    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7287c3ee-7c75-47e2-9c34-d436193f32e3" volumeName="kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389204    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3ba234a-bb37-4ed4-a505-4a8494d4e862" volumeName="kubernetes.io/secret/a3ba234a-bb37-4ed4-a505-4a8494d4e862-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389220    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" volumeName="kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389250    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0589f9a7-c2b8-408f-8374-a25b489ba69a" volumeName="kubernetes.io/configmap/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389265    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0589f9a7-c2b8-408f-8374-a25b489ba69a" volumeName="kubernetes.io/projected/0589f9a7-c2b8-408f-8374-a25b489ba69a-kube-api-access-8wvqq" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389278    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1507e7d6-a6e4-4cb9-adf5-c24d8619cede" volumeName="kubernetes.io/empty-dir/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-audit-log" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389291    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1507e7d6-a6e4-4cb9-adf5-c24d8619cede" volumeName="kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389327    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4be38679-7629-4ec8-aed1-d2ef3fcd8a10" volumeName="kubernetes.io/projected/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-kube-api-access-cxv7f" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389373    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5bb3612-89f9-4494-9634-2b8bed367954" volumeName="kubernetes.io/configmap/f5bb3612-89f9-4494-9634-2b8bed367954-config-volume" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389392    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="119a3a49-2f9f-4285-bba5-e035d61b03cd" volumeName="kubernetes.io/configmap/119a3a49-2f9f-4285-bba5-e035d61b03cd-iptables-alerter-script" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389406    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1507e7d6-a6e4-4cb9-adf5-c24d8619cede" volumeName="kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-secret-metrics-server-tls" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389420    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be73966c-8b20-4130-8203-5cf6ca09df27" volumeName="kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-bound-sa-token" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389454    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfebfdf1-70e8-4540-ba01-67df6d7fd9a9" volumeName="kubernetes.io/projected/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-kube-api-access-kcjhz" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389470    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0589f9a7-c2b8-408f-8374-a25b489ba69a" volumeName="kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389791    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5eb94f91-2be3-4a36-8d36-b3fdfb3bc026" volumeName="kubernetes.io/empty-dir/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026-tmp" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389944    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7287c3ee-7c75-47e2-9c34-d436193f32e3" volumeName="kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovnkube-config" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389965    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7287c3ee-7c75-47e2-9c34-d436193f32e3" volumeName="kubernetes.io/projected/7287c3ee-7c75-47e2-9c34-d436193f32e3-kube-api-access-c4n6f" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389978    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adb879bf-aeb5-468c-8f93-4cb40d6e4e36" volumeName="kubernetes.io/empty-dir/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-data-volume" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.389995    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" volumeName="kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cni-binary-copy" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390042    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be73966c-8b20-4130-8203-5cf6ca09df27" volumeName="kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-kube-api-access-zsbvh" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390056    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1507e7d6-a6e4-4cb9-adf5-c24d8619cede" volumeName="kubernetes.io/projected/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-kube-api-access-gmtfl" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390070    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a733526-fcbe-483b-8e2a-e58c99be5707" volumeName="kubernetes.io/projected/5a733526-fcbe-483b-8e2a-e58c99be5707-kube-api-access-h8gb9" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390093    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" volumeName="kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390137    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be73966c-8b20-4130-8203-5cf6ca09df27" volumeName="kubernetes.io/secret/be73966c-8b20-4130-8203-5cf6ca09df27-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390149    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe072ecb-08a7-4c60-ad8b-d6643882ba73" volumeName="kubernetes.io/configmap/fe072ecb-08a7-4c60-ad8b-d6643882ba73-serviceca" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390160    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59c1bd59-6c93-426a-9e81-a410a9cc62f4" volumeName="kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-config" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390176    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c53a9b9-23b0-4598-8ee6-35de80dd12d4" volumeName="kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390203    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5bb3612-89f9-4494-9634-2b8bed367954" volumeName="kubernetes.io/empty-dir/f5bb3612-89f9-4494-9634-2b8bed367954-tmp-dir" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390238    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b8fc463-aafe-4750-aaeb-827522735a5e" volumeName="kubernetes.io/configmap/0b8fc463-aafe-4750-aaeb-827522735a5e-konnectivity-ca" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390249    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1507e7d6-a6e4-4cb9-adf5-c24d8619cede" volumeName="kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-client-ca-bundle" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390260    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59c1bd59-6c93-426a-9e81-a410a9cc62f4" volumeName="kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-oauth-serving-cert" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390271    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab9d44df-91e2-4e21-afd7-eac98bf106e5" volumeName="kubernetes.io/projected/ab9d44df-91e2-4e21-afd7-eac98bf106e5-kube-api-access-httpx" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390284    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" volumeName="kubernetes.io/projected/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-kube-api-access-4jk2d" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390296    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be73966c-8b20-4130-8203-5cf6ca09df27" volumeName="kubernetes.io/configmap/be73966c-8b20-4130-8203-5cf6ca09df27-registry-certificates" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390307    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be73966c-8b20-4130-8203-5cf6ca09df27" volumeName="kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-registry-tls" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390320    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1507e7d6-a6e4-4cb9-adf5-c24d8619cede" volumeName="kubernetes.io/configmap/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390345    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59c1bd59-6c93-426a-9e81-a410a9cc62f4" volumeName="kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-service-ca" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390372    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59c1bd59-6c93-426a-9e81-a410a9cc62f4" volumeName="kubernetes.io/projected/59c1bd59-6c93-426a-9e81-a410a9cc62f4-kube-api-access-fhf7n" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390387    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7287c3ee-7c75-47e2-9c34-d436193f32e3" volumeName="kubernetes.io/secret/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390399    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3ba234a-bb37-4ed4-a505-4a8494d4e862" volumeName="kubernetes.io/projected/a3ba234a-bb37-4ed4-a505-4a8494d4e862-kube-api-access-l5tbp" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390412    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be73966c-8b20-4130-8203-5cf6ca09df27" volumeName="kubernetes.io/configmap/be73966c-8b20-4130-8203-5cf6ca09df27-trusted-ca" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390424    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be73966c-8b20-4130-8203-5cf6ca09df27" volumeName="kubernetes.io/empty-dir/be73966c-8b20-4130-8203-5cf6ca09df27-ca-trust-extracted" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390436    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b8fc463-aafe-4750-aaeb-827522735a5e" volumeName="kubernetes.io/secret/0b8fc463-aafe-4750-aaeb-827522735a5e-agent-certs" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390448    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5bb3612-89f9-4494-9634-2b8bed367954" volumeName="kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390463    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59c1bd59-6c93-426a-9e81-a410a9cc62f4" volumeName="kubernetes.io/empty-dir/59c1bd59-6c93-426a-9e81-a410a9cc62f4-tmp" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390478    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59c1bd59-6c93-426a-9e81-a410a9cc62f4" volumeName="kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-serving-cert" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390490    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a733526-fcbe-483b-8e2a-e58c99be5707" volumeName="kubernetes.io/configmap/5a733526-fcbe-483b-8e2a-e58c99be5707-cni-binary-copy" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390503    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7287c3ee-7c75-47e2-9c34-d436193f32e3" volumeName="kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-env-overrides" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390515    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95feab9f-adf7-4d34-bde6-d0a1b450df6c" volumeName="kubernetes.io/projected/95feab9f-adf7-4d34-bde6-d0a1b450df6c-kube-api-access-kvxxz" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390530    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3ba234a-bb37-4ed4-a505-4a8494d4e862" volumeName="kubernetes.io/secret/a3ba234a-bb37-4ed4-a505-4a8494d4e862-prometheus-operator-tls" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390546    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adb879bf-aeb5-468c-8f93-4cb40d6e4e36" volumeName="kubernetes.io/projected/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-api-access-2vpf2" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390558    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0589f9a7-c2b8-408f-8374-a25b489ba69a" volumeName="kubernetes.io/configmap/0589f9a7-c2b8-408f-8374-a25b489ba69a-metrics-client-ca" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390571    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a733526-fcbe-483b-8e2a-e58c99be5707" volumeName="kubernetes.io/configmap/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-daemon-config" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390583    5663 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5eb94f91-2be3-4a36-8d36-b3fdfb3bc026" volumeName="kubernetes.io/projected/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026-kube-api-access-nzvjn" seLinuxMountContext=""
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390596    5663 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 12:27:55.390960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.390605    5663 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 12:27:55.396997 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.395525    5663 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 12:27:55.426448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.426391    5663 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 12:27:55.430927 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.430896    5663 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 12:27:55.431058 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.431049    5663 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 12:27:55.431117 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.431078    5663 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 12:27:55.431177 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:27:55.431133    5663 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 12:27:55.435786 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.435733    5663 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:27:55.484985 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.484936    5663 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" containerID="b7611865f64259b6d20521cfc16ed795d43c868aa7317d03899b6dd834f4c283" exitCode=0
Oct 31 12:27:55.484985 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.484973    5663 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" containerID="098ab671dc814ca2fb8e4b7c297bdef7b8b15853b58fefe094c8c389918ff665" exitCode=0
Oct 31 12:27:55.484985 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.484984    5663 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" containerID="3dce64ab0abe72d7eaac81873d34d610fdd1d123af6cf79eba073696bb653f41" exitCode=0
Oct 31 12:27:55.484985 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.484991    5663 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" containerID="bf449d10fccf61108a1b2547dd6c0b11246077fe6ca818fbf0315bbc3498043c" exitCode=0
Oct 31 12:27:55.485238 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.484998    5663 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" containerID="7a705325516ff497eb2d443525878988a2e7226cf456979ae44d193e346771b2" exitCode=0
Oct 31 12:27:55.485238 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.485005    5663 generic.go:356] "Generic (PLEG): container finished" podID="b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a" containerID="7a39e5bb5aa773aeb4f65dfc0ec7a8e22140b61c57d04f0dc18a31b8941223c2" exitCode=0
Oct 31 12:27:55.495311 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.495272    5663 generic.go:356] "Generic (PLEG): container finished" podID="4f5b201abd7a627e01510f6952236144" containerID="702f596263a146b805d76712e1ff0addb4627fe42cea7ff86bb0b8bf29e4dd2d" exitCode=0
Oct 31 12:27:55.510130 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.510100    5663 manager.go:324] Recovery completed
Oct 31 12:27:55.513061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.512918    5663 generic.go:356] "Generic (PLEG): container finished" podID="0589f9a7-c2b8-408f-8374-a25b489ba69a" containerID="7183efda5d50ebd0b5dbd156260a12e05b67bc3a6f6424fa4a75387800fee765" exitCode=0
Oct 31 12:27:55.531246 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:27:55.531216    5663 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 12:27:55.536840 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.536805    5663 cpu_manager.go:226] "Starting" policy="none"
Oct 31 12:27:55.536840 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.536827    5663 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 12:27:55.537151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.536849    5663 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 12:27:55.537151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.537015    5663 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 12:27:55.537151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.537032    5663 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 12:27:55.537151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.537056    5663 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 12:27:55.537151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.537065    5663 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 12:27:55.537151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.537075    5663 policy_none.go:50] "Start"
Oct 31 12:27:55.537151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.537094    5663 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 12:27:55.537151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.537109    5663 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 12:27:55.538856 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.538834    5663 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 12:27:55.538953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.538862    5663 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 12:27:55.538953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.538872    5663 policy_none.go:44] "Start"
Oct 31 12:27:55.546339 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.545378    5663 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 12:27:55.554929 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.554912    5663 manager.go:341] "Starting Device Plugin manager"
Oct 31 12:27:55.555331 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:27:55.555312    5663 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 12:27:55.555467 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.555448    5663 server.go:87] "Starting device plugin registration server"
Oct 31 12:27:55.555832 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.555819    5663 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 12:27:55.556043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.556012    5663 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 12:27:55.556527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.556197    5663 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 12:27:55.556527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.556287    5663 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 12:27:55.556527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.556370    5663 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 12:27:55.556527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.556379    5663 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 12:27:55.556527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.556430    5663 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:27:55.556315632Z","UUID":"cb5da5db-05ad-4482-933f-b07dced7ed60","Handler":null,"Name":"","Endpoint":""}
Oct 31 12:27:55.558162 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:27:55.558117    5663 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 12:27:55.558656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.558635    5663 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 12:27:55.558819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.558677    5663 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 12:27:55.656840 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.656808    5663 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:27:55.659106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.659057    5663 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:27:55.659106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.659100    5663 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:27:55.659106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.659112    5663 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-137.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:27:55.659335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.659153    5663 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.673798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.673538    5663 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.673798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.673613    5663 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.732277 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.732183    5663 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"]
Oct 31 12:27:55.732927 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.732883    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal" event={"ID":"0d0c123eab79dd4d273b2febcaaf1af3","Type":"ContainerStarted","Data":"cf40578246a8f3dbd643a23b5e38641356dd7292a1342f28d4778bc317c5f242"}
Oct 31 12:27:55.733037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.732937    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal" event={"ID":"0d0c123eab79dd4d273b2febcaaf1af3","Type":"ContainerStarted","Data":"b957d88582d3c25c206d8852f928e9f9ad8952db75bc492b45f7dc21f76aaf68"}
Oct 31 12:27:55.733037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.732992    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal" event={"ID":"4f5b201abd7a627e01510f6952236144","Type":"ContainerStarted","Data":"1d591fb833f67be4150ad43177acc28cc476c050c2654d7458f9bf9ce6b90bd8"}
Oct 31 12:27:55.733037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.733011    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal" event={"ID":"4f5b201abd7a627e01510f6952236144","Type":"ContainerDied","Data":"702f596263a146b805d76712e1ff0addb4627fe42cea7ff86bb0b8bf29e4dd2d"}
Oct 31 12:27:55.733037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.733024    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal" event={"ID":"4f5b201abd7a627e01510f6952236144","Type":"ContainerStarted","Data":"08191ab633083238796092e5c3633512d3b6eb190d2f27ed6431ff7014628ede"}
Oct 31 12:27:55.733301 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.733286    5663 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.733472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.733457    5663 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.747997 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.747973    5663 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 12:27:55.748117 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:27:55.748029    5663 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.748511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.748489    5663 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 12:27:55.748580 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:27:55.748534    5663 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.796062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.795883    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f5b201abd7a627e01510f6952236144-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"4f5b201abd7a627e01510f6952236144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.796062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.795929    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5b201abd7a627e01510f6952236144-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"4f5b201abd7a627e01510f6952236144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.796062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.795959    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d0c123eab79dd4d273b2febcaaf1af3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"0d0c123eab79dd4d273b2febcaaf1af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.897115 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.897074    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d0c123eab79dd4d273b2febcaaf1af3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"0d0c123eab79dd4d273b2febcaaf1af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.897298 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.897123    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f5b201abd7a627e01510f6952236144-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"4f5b201abd7a627e01510f6952236144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.897298 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.897150    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5b201abd7a627e01510f6952236144-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"4f5b201abd7a627e01510f6952236144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.897298 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.897213    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5b201abd7a627e01510f6952236144-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"4f5b201abd7a627e01510f6952236144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.897298 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.897283    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d0c123eab79dd4d273b2febcaaf1af3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"0d0c123eab79dd4d273b2febcaaf1af3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:55.897509 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:55.897322    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f5b201abd7a627e01510f6952236144-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal\" (UID: \"4f5b201abd7a627e01510f6952236144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal"
Oct 31 12:27:56.346666 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.346628    5663 apiserver.go:52] "Watching apiserver"
Oct 31 12:27:56.356618 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.356587    5663 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 12:27:56.357282 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.357251    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pjdt2","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf","openshift-ovn-kubernetes/ovnkube-node-lnkms","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-137.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5","openshift-cluster-node-tuning-operator/tuned-5xw6s","openshift-console/console-65d5579bc6-rhzg5","openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z","kube-system/kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal","openshift-image-registry/node-ca-s9njc","openshift-dns/node-resolver-bw8kq","openshift-monitoring/metrics-server-647d9d5db7-nxnt2","openshift-network-diagnostics/network-check-target-jhwnt","openshift-image-registry/image-registry-b6467d66c-vrqf6","openshift-insights/insights-runtime-extractor-7vcws","openshift-monitoring/node-exporter-2rpgc","openshift-multus/multus-2wtl6","openshift-multus/multus-additional-cni-plugins-lxnjj","openshift-network-operator/iptables-alerter-z42qt","kube-system/global-pull-secret-syncer-p8vf7","kube-system/konnectivity-agent-blk6p","openshift-ingress-canary/ingress-canary-7lv7c","openshift-multus/network-metrics-daemon-vqlp8","openshift-console/downloads-6dd789c8b7-j5n9s"]
Oct 31 12:27:56.365302 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.365281    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 12:27:56.366874 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.366854    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 12:27:56.370968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.370934    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 12:27:56.371606 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.371586    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 12:27:56.371736 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.371722    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 12:27:56.373684 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.373660    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 12:27:56.373805 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.373768    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 12:27:56.373805 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.373787    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 12:27:56.373805 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.373798    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 12:27:56.374067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.373834    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:56.374067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.373850    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 12:27:56.374067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.373898    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bbb6f\""
Oct 31 12:27:56.374067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.373907    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 12:27:56.374067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.373930    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9d2j7\""
Oct 31 12:27:56.374067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.373976    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 12:27:56.374067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.373978    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mwkk2\""
Oct 31 12:27:56.374067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.374048    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 12:27:56.374067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.374055    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:27:56.374488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.374148    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 12:27:56.374488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.374182    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 12:27:56.374488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.374184    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 12:27:56.374488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.374252    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 12:27:56.374488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.374237    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 12:27:56.374488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.374435    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 12:27:56.375112 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.374783    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 12:27:56.375208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375130    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 12:27:56.375208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375144    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 12:27:56.375208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375153    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 12:27:56.375208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375189    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 12:27:56.375457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375231    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 12:27:56.375457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375241    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hzb2l\""
Oct 31 12:27:56.375457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375290    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kldqp\""
Oct 31 12:27:56.375457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375397    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7td2\""
Oct 31 12:27:56.375457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375443    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x5rvd\""
Oct 31 12:27:56.375697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375480    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ghjl2\""
Oct 31 12:27:56.375697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375488    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 12:27:56.375697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375520    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m5kv7\""
Oct 31 12:27:56.375697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375447    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 12:27:56.375697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375522    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-df646\""
Oct 31 12:27:56.375697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375572    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 12:27:56.375697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375641    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ng46x\""
Oct 31 12:27:56.375697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375648    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 12:27:56.376118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375763    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 12:27:56.376118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375878    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 12:27:56.376118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375924    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:56.376118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.375987    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 12:27:56.376118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376023    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcqch\""
Oct 31 12:27:56.376118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376053    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 12:27:56.376118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376065    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 12:27:56.376118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376104    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 12:27:56.376467 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376182    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 12:27:56.376467 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376200    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nzzrg\""
Oct 31 12:27:56.376467 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376267    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 12:27:56.376467 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376435    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 12:27:56.376467 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376459    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 12:27:56.376662 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376491    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-jl5lt\""
Oct 31 12:27:56.376662 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376570    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ght96\""
Oct 31 12:27:56.376811 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376789    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 12:27:56.376910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376794    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 12:27:56.376910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.376867    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8ljk6\""
Oct 31 12:27:56.377057 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.377044    5663 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 12:27:56.377278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.377265    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 12:27:56.377365 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.377267    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bpkkl\""
Oct 31 12:27:56.377419 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.377388    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 12:27:56.377477 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.377414    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 12:27:56.377737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.377723    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 12:27:56.377842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.377828    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c6kzr\""
Oct 31 12:27:56.378013 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.377990    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-llbgd\""
Oct 31 12:27:56.378110 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.378025    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 12:27:56.378110 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.378030    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 12:27:56.378742 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.378725    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 12:27:56.378822 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.378799    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 12:27:56.379551 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.379534    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 12:27:56.379597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.379542    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 12:27:56.380834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.380814    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 12:27:56.380929 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.380873    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 12:27:56.381047 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.381035    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9c5q9\""
Oct 31 12:27:56.381413 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.381396    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 12:27:56.381492 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.381422    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qx5pt\""
Oct 31 12:27:56.381492 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.381437    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 12:27:56.381492 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.381465    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 12:27:56.381492 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.381465    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9msvofqo3uh0k\""
Oct 31 12:27:56.381665 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.381528    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 12:27:56.381665 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.381556    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 12:27:56.381665 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.381601    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 12:27:56.382236 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.382217    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9nc4h\""
Oct 31 12:27:56.385493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.385476    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 12:27:56.399974 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.399952    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/119a3a49-2f9f-4285-bba5-e035d61b03cd-host-slash\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:27:56.400073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.399982    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3ba234a-bb37-4ed4-a505-4a8494d4e862-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:56.400073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400001    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-config\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.400073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400018    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-cni-multus\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.400073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400043    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-metrics-server-audit-profiles\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.400335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400101    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.400335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400141    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.400335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400165    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4n6f\" (UniqueName: \"kubernetes.io/projected/7287c3ee-7c75-47e2-9c34-d436193f32e3-kube-api-access-c4n6f\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.400335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400239    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb3612-89f9-4494-9634-2b8bed367954-config-volume\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:56.400335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400265    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-config\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.400335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400265    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3ba234a-bb37-4ed4-a505-4a8494d4e862-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:56.400634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400372    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-secret-metrics-server-client-certs\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.400634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400405    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.400634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400408    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026-tmp\") pod \"downloads-6dd789c8b7-j5n9s\" (UID: \"5eb94f91-2be3-4a36-8d36-b3fdfb3bc026\") " pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:27:56.400634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400446    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.400634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400450    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovn-node-metrics-cert\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.400634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400476    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026-tmp\") pod \"downloads-6dd789c8b7-j5n9s\" (UID: \"5eb94f91-2be3-4a36-8d36-b3fdfb3bc026\") " pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:27:56.400634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400509    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3ba234a-bb37-4ed4-a505-4a8494d4e862-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:56.400634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400532    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb3612-89f9-4494-9634-2b8bed367954-config-volume\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:56.400634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400550    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-tls\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.400634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400568    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-oauth-serving-cert\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.400634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400588    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be73966c-8b20-4130-8203-5cf6ca09df27-registry-certificates\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.400634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400584    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-secret-metrics-server-client-certs\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400641    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/542d428b-b94d-4033-9805-d13da84ede85-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qzkwf\" (UID: \"542d428b-b94d-4033-9805-d13da84ede85\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400672    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-system-cni-dir\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400699    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-service-ca\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400724    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-tmp\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400746    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3ba234a-bb37-4ed4-a505-4a8494d4e862-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400749    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-bound-sa-token\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400766    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-oauth-serving-cert\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400788    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-tls\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400825    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400854    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-textfile\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400859    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovn-node-metrics-cert\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400880    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvqq\" (UniqueName: \"kubernetes.io/projected/0589f9a7-c2b8-408f-8374-a25b489ba69a-kube-api-access-8wvqq\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400890    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be73966c-8b20-4130-8203-5cf6ca09df27-registry-certificates\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400972    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-service-ca\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400986    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/542d428b-b94d-4033-9805-d13da84ede85-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qzkwf\" (UID: \"542d428b-b94d-4033-9805-d13da84ede85\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400990    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-textfile\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401098    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.400908    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be73966c-8b20-4130-8203-5cf6ca09df27-installation-pull-secrets\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.401177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401176    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l822j\" (UniqueName: \"kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j\") pod \"network-check-target-jhwnt\" (UID: \"23cc7cc6-1673-434b-a56f-e83298dc5d91\") " pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401195    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysctl-conf\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401205    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be73966c-8b20-4130-8203-5cf6ca09df27-installation-pull-secrets\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401210    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-node-log\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401226    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0589f9a7-c2b8-408f-8374-a25b489ba69a-metrics-client-ca\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401245    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-serving-cert\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401284    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-modprobe-d\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401300    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-tuned\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401315    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysconfig\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401347    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-conf-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401394    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be73966c-8b20-4130-8203-5cf6ca09df27-image-registry-private-configuration\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401420    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmtfl\" (UniqueName: \"kubernetes.io/projected/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-kube-api-access-gmtfl\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401417    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-tuned\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401463    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-etc-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401486    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-netns\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401518    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-multus-certs\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401544    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bb3612-89f9-4494-9634-2b8bed367954-tmp-dir\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401557    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0589f9a7-c2b8-408f-8374-a25b489ba69a-metrics-client-ca\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401560    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-serving-cert\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401578    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-sys\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401622    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bb3612-89f9-4494-9634-2b8bed367954-tmp-dir\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401654    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9d44df-91e2-4e21-afd7-eac98bf106e5-tmp-dir\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401670    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-tmp\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401708    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be73966c-8b20-4130-8203-5cf6ca09df27-image-registry-private-configuration\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401697    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-socket-dir-parent\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401734    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab9d44df-91e2-4e21-afd7-eac98bf106e5-tmp-dir\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401746    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-cni-bin\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401776    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401798    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401828    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-cni-bin\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401866    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-root\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401889    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-device-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401919    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxxz\" (UniqueName: \"kubernetes.io/projected/95feab9f-adf7-4d34-bde6-d0a1b450df6c-kube-api-access-kvxxz\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401956    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-log-socket\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.401993    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-cni-netd\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402034    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovnkube-script-lib\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.402142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402068    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysctl-d\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402250    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-sys\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402270    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-cni-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402286    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-registration-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402294    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovnkube-script-lib\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402317    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5tbp\" (UniqueName: \"kubernetes.io/projected/a3ba234a-bb37-4ed4-a505-4a8494d4e862-kube-api-access-l5tbp\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402339    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402397    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cnibin\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402453    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzvjn\" (UniqueName: \"kubernetes.io/projected/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026-kube-api-access-nzvjn\") pod \"downloads-6dd789c8b7-j5n9s\" (UID: \"5eb94f91-2be3-4a36-8d36-b3fdfb3bc026\") " pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402563    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402575    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402616    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402645    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-secret-metrics-server-tls\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402672    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-run\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402716    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402745    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe072ecb-08a7-4c60-ad8b-d6643882ba73-serviceca\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402773    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4958\" (UniqueName: \"kubernetes.io/projected/fe072ecb-08a7-4c60-ad8b-d6643882ba73-kube-api-access-m4958\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402914    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-secret-metrics-server-tls\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402920    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402903    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402964    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-original-pull-secret\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402974    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-kubernetes\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402982    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe072ecb-08a7-4c60-ad8b-d6643882ba73-serviceca\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.402999    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-kubelet\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403033    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-daemon-config\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403076    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-lib-modules\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403106    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-system-cni-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403151    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403164    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-daemon-config\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403180    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403207    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-slash\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403250    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-oauth-config\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403422    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5bb3612-89f9-4494-9634-2b8bed367954-metrics-tls\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403340    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403470    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-env-overrides\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403498    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-audit-log\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403523    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-client-ca-bundle\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403547    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-registry-tls\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403571    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxv7f\" (UniqueName: \"kubernetes.io/projected/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-kube-api-access-cxv7f\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403591    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-audit-log\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403599    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-crio-socket\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403654    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-env-overrides\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.403641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403652    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-kubelet-config\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403691    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-dbus\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403733    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3ba234a-bb37-4ed4-a505-4a8494d4e862-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403765    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhf7n\" (UniqueName: \"kubernetes.io/projected/59c1bd59-6c93-426a-9e81-a410a9cc62f4-kube-api-access-fhf7n\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403839    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-registry-tls\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403885    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-etc-kubernetes\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403907    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-data-volume\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403924    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-client-ca-bundle\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403942    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403964    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3ba234a-bb37-4ed4-a505-4a8494d4e862-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.403990    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovnkube-config\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404005    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-data-volume\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404020    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vpf2\" (UniqueName: \"kubernetes.io/projected/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-api-access-2vpf2\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404037    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jk2d\" (UniqueName: \"kubernetes.io/projected/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-kube-api-access-4jk2d\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404060    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-kubelet\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404077    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-systemd-units\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404091    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/609830fb-fa9a-4886-89be-cb40635b2c67-cert\") pod \"ingress-canary-7lv7c\" (UID: \"609830fb-fa9a-4886-89be-cb40635b2c67\") " pod="openshift-ingress-canary/ingress-canary-7lv7c"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404108    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-socket-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404161    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404212    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7287c3ee-7c75-47e2-9c34-d436193f32e3-ovnkube-config\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404225    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404265    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-ovn\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404295    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be73966c-8b20-4130-8203-5cf6ca09df27-trusted-ca\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404348    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/609830fb-fa9a-4886-89be-cb40635b2c67-cert\") pod \"ingress-canary-7lv7c\" (UID: \"609830fb-fa9a-4886-89be-cb40635b2c67\") " pod="openshift-ingress-canary/ingress-canary-7lv7c"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404410    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8fc463-aafe-4750-aaeb-827522735a5e-agent-certs\") pod \"konnectivity-agent-blk6p\" (UID: \"0b8fc463-aafe-4750-aaeb-827522735a5e\") " pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404510    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95feab9f-adf7-4d34-bde6-d0a1b450df6c-metrics-certs\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404533    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8fc463-aafe-4750-aaeb-827522735a5e-konnectivity-ca\") pod \"konnectivity-agent-blk6p\" (UID: \"0b8fc463-aafe-4750-aaeb-827522735a5e\") " pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404578    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cni-binary-copy\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404606    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-hostroot\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404644    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2shhr\" (UniqueName: \"kubernetes.io/projected/609830fb-fa9a-4886-89be-cb40635b2c67-kube-api-access-2shhr\") pod \"ingress-canary-7lv7c\" (UID: \"609830fb-fa9a-4886-89be-cb40635b2c67\") " pod="openshift-ingress-canary/ingress-canary-7lv7c"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404715    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8fc463-aafe-4750-aaeb-827522735a5e-konnectivity-ca\") pod \"konnectivity-agent-blk6p\" (UID: \"0b8fc463-aafe-4750-aaeb-827522735a5e\") " pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404717    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be73966c-8b20-4130-8203-5cf6ca09df27-trusted-ca\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404745    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab9d44df-91e2-4e21-afd7-eac98bf106e5-hosts-file\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404767    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-systemd\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404784    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-trusted-ca-bundle\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404798    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a733526-fcbe-483b-8e2a-e58c99be5707-cni-binary-copy\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404814    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-k8s-cni-cncf-io\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404837    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8fc463-aafe-4750-aaeb-827522735a5e-agent-certs\") pod \"konnectivity-agent-blk6p\" (UID: \"0b8fc463-aafe-4750-aaeb-827522735a5e\") " pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404851    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8gb9\" (UniqueName: \"kubernetes.io/projected/5a733526-fcbe-483b-8e2a-e58c99be5707-kube-api-access-h8gb9\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404854    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cni-binary-copy\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404895    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4j49\" (UniqueName: \"kubernetes.io/projected/f5bb3612-89f9-4494-9634-2b8bed367954-kube-api-access-c4j49\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.404940    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe072ecb-08a7-4c60-ad8b-d6643882ba73-host\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405025    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a733526-fcbe-483b-8e2a-e58c99be5707-cni-binary-copy\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405038    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-wtmp\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405116    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-trusted-ca-bundle\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405151    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcjhz\" (UniqueName: \"kubernetes.io/projected/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-kube-api-access-kcjhz\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405175    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-cnibin\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.405210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405194    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httpx\" (UniqueName: \"kubernetes.io/projected/ab9d44df-91e2-4e21-afd7-eac98bf106e5-kube-api-access-httpx\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405237    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gljhb\" (UniqueName: \"kubernetes.io/projected/119a3a49-2f9f-4285-bba5-e035d61b03cd-kube-api-access-gljhb\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405272    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-systemd\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405297    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-run-netns\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405333    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-var-lib-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405380    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c1bd59-6c93-426a-9e81-a410a9cc62f4-tmp\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405407    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-var-lib-kubelet\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405434    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbvh\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-kube-api-access-zsbvh\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405486    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c1bd59-6c93-426a-9e81-a410a9cc62f4-tmp\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405514    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-sys-fs\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405597    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/119a3a49-2f9f-4285-bba5-e035d61b03cd-iptables-alerter-script\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405615    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-host\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405629    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be73966c-8b20-4130-8203-5cf6ca09df27-ca-trust-extracted\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405644    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405744    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be73966c-8b20-4130-8203-5cf6ca09df27-ca-trust-extracted\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405836    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.406572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.405930    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/119a3a49-2f9f-4285-bba5-e035d61b03cd-iptables-alerter-script\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:27:56.416698 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.416677    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4n6f\" (UniqueName: \"kubernetes.io/projected/7287c3ee-7c75-47e2-9c34-d436193f32e3-kube-api-access-c4n6f\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.417775 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.417757    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5tbp\" (UniqueName: \"kubernetes.io/projected/a3ba234a-bb37-4ed4-a505-4a8494d4e862-kube-api-access-l5tbp\") pod \"prometheus-operator-9cd89b74c-ptf8z\" (UID: \"a3ba234a-bb37-4ed4-a505-4a8494d4e862\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ptf8z"
Oct 31 12:27:56.417838 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.417797    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-bound-sa-token\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.418461 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.418447    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzvjn\" (UniqueName: \"kubernetes.io/projected/5eb94f91-2be3-4a36-8d36-b3fdfb3bc026-kube-api-access-nzvjn\") pod \"downloads-6dd789c8b7-j5n9s\" (UID: \"5eb94f91-2be3-4a36-8d36-b3fdfb3bc026\") " pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:27:56.419686 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.419669    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxxz\" (UniqueName: \"kubernetes.io/projected/95feab9f-adf7-4d34-bde6-d0a1b450df6c-kube-api-access-kvxxz\") pod \"network-metrics-daemon-vqlp8\" (UID: \"95feab9f-adf7-4d34-bde6-d0a1b450df6c\") " pod="openshift-multus/network-metrics-daemon-vqlp8"
Oct 31 12:27:56.419735 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.419717    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gljhb\" (UniqueName: \"kubernetes.io/projected/119a3a49-2f9f-4285-bba5-e035d61b03cd-kube-api-access-gljhb\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:27:56.419977 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.419961    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jk2d\" (UniqueName: \"kubernetes.io/projected/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-kube-api-access-4jk2d\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.420588 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.420571    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmtfl\" (UniqueName: \"kubernetes.io/projected/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-kube-api-access-gmtfl\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.423126 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.423110    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l822j\" (UniqueName: \"kubernetes.io/projected/23cc7cc6-1673-434b-a56f-e83298dc5d91-kube-api-access-l822j\") pod \"network-check-target-jhwnt\" (UID: \"23cc7cc6-1673-434b-a56f-e83298dc5d91\") " pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:27:56.435695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.435677    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxv7f\" (UniqueName: \"kubernetes.io/projected/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-kube-api-access-cxv7f\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.435986 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.435965    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4j49\" (UniqueName: \"kubernetes.io/projected/f5bb3612-89f9-4494-9634-2b8bed367954-kube-api-access-c4j49\") pod \"dns-default-pjdt2\" (UID: \"f5bb3612-89f9-4494-9634-2b8bed367954\") " pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:56.436137 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.436118    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-httpx\" (UniqueName: \"kubernetes.io/projected/ab9d44df-91e2-4e21-afd7-eac98bf106e5-kube-api-access-httpx\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:27:56.436175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.436139    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhf7n\" (UniqueName: \"kubernetes.io/projected/59c1bd59-6c93-426a-9e81-a410a9cc62f4-kube-api-access-fhf7n\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.438520 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.438503    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2shhr\" (UniqueName: \"kubernetes.io/projected/609830fb-fa9a-4886-89be-cb40635b2c67-kube-api-access-2shhr\") pod \"ingress-canary-7lv7c\" (UID: \"609830fb-fa9a-4886-89be-cb40635b2c67\") " pod="openshift-ingress-canary/ingress-canary-7lv7c"
Oct 31 12:27:56.439116 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.439101    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvqq\" (UniqueName: \"kubernetes.io/projected/0589f9a7-c2b8-408f-8374-a25b489ba69a-kube-api-access-8wvqq\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.439179 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.439164    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbvh\" (UniqueName: \"kubernetes.io/projected/be73966c-8b20-4130-8203-5cf6ca09df27-kube-api-access-zsbvh\") pod \"image-registry-b6467d66c-vrqf6\" (UID: \"be73966c-8b20-4130-8203-5cf6ca09df27\") " pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:56.442279 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.442262    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8gb9\" (UniqueName: \"kubernetes.io/projected/5a733526-fcbe-483b-8e2a-e58c99be5707-kube-api-access-h8gb9\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.442331 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.442316    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcjhz\" (UniqueName: \"kubernetes.io/projected/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-kube-api-access-kcjhz\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.442401 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.442387    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4958\" (UniqueName: \"kubernetes.io/projected/fe072ecb-08a7-4c60-ad8b-d6643882ba73-kube-api-access-m4958\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:27:56.442437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.442421    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vpf2\" (UniqueName: \"kubernetes.io/projected/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-kube-api-access-2vpf2\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:56.464168 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.464137    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 12:27:56.465641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.465626    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 12:27:56.470933 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.470916    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1507e7d6-a6e4-4cb9-adf5-c24d8619cede-metrics-server-audit-profiles\") pod \"metrics-server-647d9d5db7-nxnt2\" (UID: \"1507e7d6-a6e4-4cb9-adf5-c24d8619cede\") " pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:27:56.474133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.474113    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-oauth-config\") pod \"console-65d5579bc6-rhzg5\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") " pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:27:56.506327 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506298    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-cni-multus\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.506493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506333    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-system-cni-dir\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.506493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506366    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysctl-conf\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.506493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506420    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-system-cni-dir\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.506493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506422    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-cni-multus\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.506493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506470    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-node-log\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.506732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506508    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-modprobe-d\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.506732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506514    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-node-log\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.506732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506545    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysconfig\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.506732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506571    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-conf-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.506732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506598    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-etc-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.506732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506636    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysconfig\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.506732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506641    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-modprobe-d\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.506732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506644    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-etc-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.506732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506647    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-conf-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.506732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506682    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysctl-conf\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.506732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506702    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-netns\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.506732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506722    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-multus-certs\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506740    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-sys\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506757    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-socket-dir-parent\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506806    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-netns\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506840    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-multus-certs\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506885    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-socket-dir-parent\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506908    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-sys\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506956    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-cni-bin\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506985    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.506993    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-cni-bin\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507011    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507041    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-cni-bin\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507064    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507043    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507063    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-root\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507090    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-root\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507098    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-cni-bin\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507128    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-device-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507155    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-log-socket\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507173    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-cni-netd\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507189    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-device-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507194    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysctl-d\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507209    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-sys\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507222    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-cni-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507227    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-log-socket\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507234    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-cni-netd\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507237    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-registration-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507275    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cnibin\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507289    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507290    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-sys\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507301    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-sysctl-d\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507304    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-run\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507335    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507338    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-run\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507337    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507301    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-multus-cni-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507363    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-kubernetes\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507365    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-registration-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507388    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-kubelet\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507406    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.507392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507415    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-kubernetes\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507413    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-lib-modules\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507452    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-system-cni-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507466    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-var-lib-kubelet\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507504    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-lib-modules\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507505    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507522    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-system-cni-dir\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507537    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-slash\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507546    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507558    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a-cnibin\") pod \"multus-additional-cni-plugins-lxnjj\" (UID: \"b9e1ad7d-c64f-4f2b-b14a-c9b69d97f43a\") " pod="openshift-multus/multus-additional-cni-plugins-lxnjj"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507566    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507591    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-crio-socket\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507606    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-kubelet-config\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507608    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-slash\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507637    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-kubelet-config\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507638    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507653    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-dbus\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507666    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/adb879bf-aeb5-468c-8f93-4cb40d6e4e36-crio-socket\") pod \"insights-runtime-extractor-7vcws\" (UID: \"adb879bf-aeb5-468c-8f93-4cb40d6e4e36\") " pod="openshift-insights/insights-runtime-extractor-7vcws"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507687    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-etc-kubernetes\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507710    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-kubelet\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507720    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c53a9b9-23b0-4598-8ee6-35de80dd12d4-dbus\") pod \"global-pull-secret-syncer-p8vf7\" (UID: \"8c53a9b9-23b0-4598-8ee6-35de80dd12d4\") " pod="kube-system/global-pull-secret-syncer-p8vf7"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507734    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-kubelet\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507734    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-etc-kubernetes\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507745    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-systemd-units\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507761    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-socket-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507786    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-ovn\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507792    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-systemd-units\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507804    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-hostroot\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507826    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-ovn\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507832    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-hostroot\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507843    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab9d44df-91e2-4e21-afd7-eac98bf106e5-hosts-file\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507871    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-systemd\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507843    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-socket-dir\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507888    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab9d44df-91e2-4e21-afd7-eac98bf106e5-hosts-file\") pod \"node-resolver-bw8kq\" (UID: \"ab9d44df-91e2-4e21-afd7-eac98bf106e5\") " pod="openshift-dns/node-resolver-bw8kq"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507896    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-k8s-cni-cncf-io\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507941    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-host-run-k8s-cni-cncf-io\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507953    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe072ecb-08a7-4c60-ad8b-d6643882ba73-host\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507922    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-run-systemd\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507970    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-wtmp\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.507985    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-cnibin\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508007    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-systemd\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508009    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe072ecb-08a7-4c60-ad8b-d6643882ba73-host\") pod \"node-ca-s9njc\" (UID: \"fe072ecb-08a7-4c60-ad8b-d6643882ba73\") " pod="openshift-image-registry/node-ca-s9njc"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508024    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-run-netns\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508039    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-var-lib-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508042    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a733526-fcbe-483b-8e2a-e58c99be5707-cnibin\") pod \"multus-2wtl6\" (UID: \"5a733526-fcbe-483b-8e2a-e58c99be5707\") " pod="openshift-multus/multus-2wtl6"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508009    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0589f9a7-c2b8-408f-8374-a25b489ba69a-node-exporter-wtmp\") pod \"node-exporter-2rpgc\" (UID: \"0589f9a7-c2b8-408f-8374-a25b489ba69a\") " pod="openshift-monitoring/node-exporter-2rpgc"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508065    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-etc-systemd\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508066    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-var-lib-kubelet\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508088    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-var-lib-openvswitch\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508099    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-sys-fs\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508104    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-var-lib-kubelet\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508122    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7287c3ee-7c75-47e2-9c34-d436193f32e3-host-run-netns\") pod \"ovnkube-node-lnkms\" (UID: \"7287c3ee-7c75-47e2-9c34-d436193f32e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508126    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-host\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508159    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4be38679-7629-4ec8-aed1-d2ef3fcd8a10-sys-fs\") pod \"aws-ebs-csi-driver-node-m9bq5\" (UID: \"4be38679-7629-4ec8-aed1-d2ef3fcd8a10\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9bq5"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508161    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/119a3a49-2f9f-4285-bba5-e035d61b03cd-host-slash\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508188    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfebfdf1-70e8-4540-ba01-67df6d7fd9a9-host\") pod \"tuned-5xw6s\" (UID: \"bfebfdf1-70e8-4540-ba01-67df6d7fd9a9\") " pod="openshift-cluster-node-tuning-operator/tuned-5xw6s"
Oct 31 12:27:56.508996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.508230    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/119a3a49-2f9f-4285-bba5-e035d61b03cd-host-slash\") pod \"iptables-alerter-z42qt\" (UID: \"119a3a49-2f9f-4285-bba5-e035d61b03cd\") " pod="openshift-network-operator/iptables-alerter-z42qt"
Oct 31 12:27:56.666910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:56.666886    5663 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:27:58.299209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:58.299170    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pjdt2"
Oct 31 12:27:58.546817 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:58.546586    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j5n9s" event={"ID":"5eb94f91-2be3-4a36-8d36-b3fdfb3bc026","Type":"ContainerStarted","Data":"1428209a98d413dc4a08141aea7e92d7fa84ea0f68773436c66ca1d7255315b4"}
Oct 31 12:27:58.548643 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:58.548610    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d5579bc6-rhzg5" event={"ID":"59c1bd59-6c93-426a-9e81-a410a9cc62f4","Type":"ContainerStarted","Data":"1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321"}
Oct 31 12:27:58.561832 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:58.561807    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:58.682519 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:58.682487    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b6467d66c-vrqf6"
Oct 31 12:27:59.284589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:59.284557    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:27:59.552562 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:59.552489    5663 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:27:59.780946 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:27:59.780908    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qzkwf"
Oct 31 12:28:00.756251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:00.756212    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:28:00.761781 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:00.761754    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:28:01.006061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:01.006016    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:28:03.916710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:03.916619    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-blk6p"
Oct 31 12:28:04.569206 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:04.569112    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:28:04.569466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:04.569276    5663 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:28:04.574560 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:04.574532    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:28:06.406460 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:06.406423    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:28:06.406887 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:06.406546    5663 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:28:06.410288 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:06.410262    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-j5n9s"
Oct 31 12:28:12.568144 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:12.568097    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:28:13.135747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:13.135713    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:28:13.904178 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:13.904150    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnkms"
Oct 31 12:28:22.474540 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:28:22.474502    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:28:23.705909 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.705879    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c46f6c557-tc9hn"]
Oct 31 12:28:23.740310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.740277    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c46f6c557-tc9hn"]
Oct 31 12:28:23.740471 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.740431    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.744408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.744379    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 12:28:23.747529 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.747501    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 12:28:23.804199 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.804172    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 12:28:23.804199 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.804206    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 12:28:23.804442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.804393    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 12:28:23.804442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.804398    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ds9m7\""
Oct 31 12:28:23.804442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.804427    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 12:28:23.835136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.834863    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be55aab1-5de3-442e-9f57-bc268d796440-serving-certs-ca-bundle\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.835136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.834914    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/be55aab1-5de3-442e-9f57-bc268d796440-secret-telemeter-client\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.835136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.834944    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/be55aab1-5de3-442e-9f57-bc268d796440-telemeter-client-tls\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.835136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.834978    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be55aab1-5de3-442e-9f57-bc268d796440-metrics-client-ca\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.835136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.835003    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be55aab1-5de3-442e-9f57-bc268d796440-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.835136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.835045    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275h4\" (UniqueName: \"kubernetes.io/projected/be55aab1-5de3-442e-9f57-bc268d796440-kube-api-access-275h4\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.835489 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.835235    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/be55aab1-5de3-442e-9f57-bc268d796440-federate-client-tls\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.835489 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.835271    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be55aab1-5de3-442e-9f57-bc268d796440-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.937496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.937447    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/be55aab1-5de3-442e-9f57-bc268d796440-federate-client-tls\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.937692 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.937511    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be55aab1-5de3-442e-9f57-bc268d796440-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.937692 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.937562    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be55aab1-5de3-442e-9f57-bc268d796440-serving-certs-ca-bundle\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.937692 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.937616    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/be55aab1-5de3-442e-9f57-bc268d796440-secret-telemeter-client\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.937692 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.937649    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/be55aab1-5de3-442e-9f57-bc268d796440-telemeter-client-tls\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.937692 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.937689    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be55aab1-5de3-442e-9f57-bc268d796440-metrics-client-ca\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.937930 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.937718    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be55aab1-5de3-442e-9f57-bc268d796440-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.937930 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.937755    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275h4\" (UniqueName: \"kubernetes.io/projected/be55aab1-5de3-442e-9f57-bc268d796440-kube-api-access-275h4\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.938001 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.937950    5663 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 12:28:23.938624 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.938563    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be55aab1-5de3-442e-9f57-bc268d796440-metrics-client-ca\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.938989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.938959    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be55aab1-5de3-442e-9f57-bc268d796440-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.939371 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.939334    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be55aab1-5de3-442e-9f57-bc268d796440-serving-certs-ca-bundle\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.940507 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.940464    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be55aab1-5de3-442e-9f57-bc268d796440-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.940751 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.940734    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/be55aab1-5de3-442e-9f57-bc268d796440-federate-client-tls\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.940817 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.940776    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/be55aab1-5de3-442e-9f57-bc268d796440-telemeter-client-tls\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.940914 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.940898    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/be55aab1-5de3-442e-9f57-bc268d796440-secret-telemeter-client\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:23.950248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:23.950219    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275h4\" (UniqueName: \"kubernetes.io/projected/be55aab1-5de3-442e-9f57-bc268d796440-kube-api-access-275h4\") pod \"telemeter-client-c46f6c557-tc9hn\" (UID: \"be55aab1-5de3-442e-9f57-bc268d796440\") " pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:24.092632 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:24.092595    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn"
Oct 31 12:28:24.402909 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:24.402826    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c46f6c557-tc9hn"]
Oct 31 12:28:24.407075 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:28:24.407040    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe55aab1_5de3_442e_9f57_bc268d796440.slice/crio-11ffdfe75fa35298380da15d57313fd88f3fe94eed4059d88411bea531092648 WatchSource:0}: Error finding container 11ffdfe75fa35298380da15d57313fd88f3fe94eed4059d88411bea531092648: Status 404 returned error can't find the container with id 11ffdfe75fa35298380da15d57313fd88f3fe94eed4059d88411bea531092648
Oct 31 12:28:24.635677 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:24.635639    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn" event={"ID":"be55aab1-5de3-442e-9f57-bc268d796440","Type":"ContainerStarted","Data":"11ffdfe75fa35298380da15d57313fd88f3fe94eed4059d88411bea531092648"}
Oct 31 12:28:29.654495 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:29.654460    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn" event={"ID":"be55aab1-5de3-442e-9f57-bc268d796440","Type":"ContainerStarted","Data":"f9c05f85da8d62ebe7d44fd865f1edf504c852f2cdf797ef14d0fdfc7830a687"}
Oct 31 12:28:31.009457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.009429    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jhwnt"
Oct 31 12:28:31.661742 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.661702    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn" event={"ID":"be55aab1-5de3-442e-9f57-bc268d796440","Type":"ContainerStarted","Data":"f4fff5b1a70967b76f77fd5ea0d892f0e52d954df145616a503deca61182112a"}
Oct 31 12:28:31.661911 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.661754    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn" event={"ID":"be55aab1-5de3-442e-9f57-bc268d796440","Type":"ContainerStarted","Data":"ee44b4f0f115ce337573cc423f1ac0b52955346688629fd97d68c1226a0fe06a"}
Oct 31 12:28:31.687677 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.687624    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c46f6c557-tc9hn" podStartSLOduration=2.40478315 podStartE2EDuration="8.687611042s" totalImagesPullingTime="6.282827892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:28:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:28:31.686664601 +0000 UTC m=+36.427090817" watchObservedRunningTime="2026-10-31 12:28:31.687611042 +0000 UTC m=+36.428037255"
Oct 31 12:28:31.779867 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.779835    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69d674879f-l9xgc"]
Oct 31 12:28:31.781668 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.781651    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:31.793303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.793253    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d674879f-l9xgc"]
Oct 31 12:28:31.903953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.903915    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-oauth-serving-cert\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:31.904127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.903962    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-trusted-ca-bundle\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:31.904127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.904031    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-serving-cert\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:31.904127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.904095    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-console-config\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:31.904284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.904159    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-service-ca\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:31.904284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.904217    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10ea48b-480f-4d01-8884-73286c7cb255-tmp\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:31.904284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.904272    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-oauth-config\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:31.904444 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:31.904309    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctqf\" (UniqueName: \"kubernetes.io/projected/a10ea48b-480f-4d01-8884-73286c7cb255-kube-api-access-9ctqf\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.005311 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.005250    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-console-config\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.005311 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.005284    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-service-ca\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.005311 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.005301    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10ea48b-480f-4d01-8884-73286c7cb255-tmp\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.005533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.005435    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-oauth-config\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.005533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.005475    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctqf\" (UniqueName: \"kubernetes.io/projected/a10ea48b-480f-4d01-8884-73286c7cb255-kube-api-access-9ctqf\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.005533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.005511    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-oauth-serving-cert\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.005671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.005547    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-trusted-ca-bundle\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.005671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.005573    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-serving-cert\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.005804 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.005784    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10ea48b-480f-4d01-8884-73286c7cb255-tmp\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.006299 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.006272    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-oauth-serving-cert\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.006299 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.006288    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-console-config\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.006472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.006273    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-service-ca\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.006691 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.006672    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-trusted-ca-bundle\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.007859 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.007836    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-oauth-config\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.008011 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.007992    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-serving-cert\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.017545 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.017525    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctqf\" (UniqueName: \"kubernetes.io/projected/a10ea48b-480f-4d01-8884-73286c7cb255-kube-api-access-9ctqf\") pod \"console-69d674879f-l9xgc\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") " pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.091950 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.091917    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:32.362068 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.362034    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d674879f-l9xgc"]
Oct 31 12:28:32.365441 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:28:32.365415    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10ea48b_480f_4d01_8884_73286c7cb255.slice/crio-efe265b98f18c6b6bbe7f1f2a45de06ca75eaa2559209a33dd103be78eeebc7d WatchSource:0}: Error finding container efe265b98f18c6b6bbe7f1f2a45de06ca75eaa2559209a33dd103be78eeebc7d: Status 404 returned error can't find the container with id efe265b98f18c6b6bbe7f1f2a45de06ca75eaa2559209a33dd103be78eeebc7d
Oct 31 12:28:32.572594 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.572563    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:28:32.576404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.576380    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-647d9d5db7-nxnt2"
Oct 31 12:28:32.667402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.667304    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d674879f-l9xgc" event={"ID":"a10ea48b-480f-4d01-8884-73286c7cb255","Type":"ContainerStarted","Data":"1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14"}
Oct 31 12:28:32.667402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.667378    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d674879f-l9xgc" event={"ID":"a10ea48b-480f-4d01-8884-73286c7cb255","Type":"ContainerStarted","Data":"efe265b98f18c6b6bbe7f1f2a45de06ca75eaa2559209a33dd103be78eeebc7d"}
Oct 31 12:28:32.688798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:32.688743    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69d674879f-l9xgc" podStartSLOduration=1.688728129 podStartE2EDuration="1.688728129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:28:32.687984462 +0000 UTC m=+37.428410677" watchObservedRunningTime="2026-10-31 12:28:32.688728129 +0000 UTC m=+37.429154346"
Oct 31 12:28:42.093060 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:42.093017    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:42.093576 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:42.093097    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:42.097731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:42.097708    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:42.705968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:42.705940    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:28:42.755259 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:28:42.755225    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65d5579bc6-rhzg5"]
Oct 31 12:28:52.474664 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:28:52.474623    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:29:07.774933 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:07.774872    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-65d5579bc6-rhzg5" podUID="59c1bd59-6c93-426a-9e81-a410a9cc62f4" containerName="console" containerID="cri-o://1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321" gracePeriod=15
Oct 31 12:29:08.029649 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.029586    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:29:08.137629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.137592    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-555fc688cb-25l7p"]
Oct 31 12:29:08.137905 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.137891    5663 cpu_manager.go:414] "containerMap: removing container" podUID="59c1bd59-6c93-426a-9e81-a410a9cc62f4" containerName="console"
Oct 31 12:29:08.137944 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.137907    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c1bd59-6c93-426a-9e81-a410a9cc62f4" containerName="console"
Oct 31 12:29:08.137977 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.137962    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c1bd59-6c93-426a-9e81-a410a9cc62f4" containerName="console"
Oct 31 12:29:08.141054 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.141039    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.151425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.151399    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-555fc688cb-25l7p"]
Oct 31 12:29:08.203542 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.203506    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-trusted-ca-bundle\") pod \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") "
Oct 31 12:29:08.203731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.203555    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-oauth-config\") pod \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") "
Oct 31 12:29:08.203731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.203596    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-config\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-config\") pod \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") "
Oct 31 12:29:08.203731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.203614    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c1bd59-6c93-426a-9e81-a410a9cc62f4-kube-api-access-fhf7n\" (UniqueName: \"kubernetes.io/projected/59c1bd59-6c93-426a-9e81-a410a9cc62f4-kube-api-access-fhf7n\") pod \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") "
Oct 31 12:29:08.203731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.203643    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-service-ca\") pod \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") "
Oct 31 12:29:08.203731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.203709    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-serving-cert\") pod \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") "
Oct 31 12:29:08.203994 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.203744    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-oauth-serving-cert\") pod \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") "
Oct 31 12:29:08.203994 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.203773    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c1bd59-6c93-426a-9e81-a410a9cc62f4-tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c1bd59-6c93-426a-9e81-a410a9cc62f4-tmp\") pod \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\" (UID: \"59c1bd59-6c93-426a-9e81-a410a9cc62f4\") "
Oct 31 12:29:08.204101 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.204050    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-config" pod "59c1bd59-6c93-426a-9e81-a410a9cc62f4" (UID: "59c1bd59-6c93-426a-9e81-a410a9cc62f4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:29:08.204101 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.204071    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-trusted-ca-bundle" pod "59c1bd59-6c93-426a-9e81-a410a9cc62f4" (UID: "59c1bd59-6c93-426a-9e81-a410a9cc62f4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:29:08.204236 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.204215    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c1bd59-6c93-426a-9e81-a410a9cc62f4-tmp" pod "59c1bd59-6c93-426a-9e81-a410a9cc62f4" (UID: "59c1bd59-6c93-426a-9e81-a410a9cc62f4"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:29:08.204288 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.204239    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-oauth-serving-cert" pod "59c1bd59-6c93-426a-9e81-a410a9cc62f4" (UID: "59c1bd59-6c93-426a-9e81-a410a9cc62f4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:29:08.204469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.204440    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-service-ca" pod "59c1bd59-6c93-426a-9e81-a410a9cc62f4" (UID: "59c1bd59-6c93-426a-9e81-a410a9cc62f4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:29:08.206062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.206033    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c1bd59-6c93-426a-9e81-a410a9cc62f4-kube-api-access-fhf7n" pod "59c1bd59-6c93-426a-9e81-a410a9cc62f4" (UID: "59c1bd59-6c93-426a-9e81-a410a9cc62f4"). InnerVolumeSpecName "kube-api-access-fhf7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:29:08.206062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.206039    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-oauth-config" pod "59c1bd59-6c93-426a-9e81-a410a9cc62f4" (UID: "59c1bd59-6c93-426a-9e81-a410a9cc62f4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:29:08.206186 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.206122    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-serving-cert" pod "59c1bd59-6c93-426a-9e81-a410a9cc62f4" (UID: "59c1bd59-6c93-426a-9e81-a410a9cc62f4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:29:08.304613 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304528    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f869ae4c-2b69-48e5-861b-425036ebe3b8-tmp\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.304613 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304577    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f869ae4c-2b69-48e5-861b-425036ebe3b8-console-oauth-config\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.304802 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304636    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f869ae4c-2b69-48e5-861b-425036ebe3b8-trusted-ca-bundle\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.304802 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304665    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f869ae4c-2b69-48e5-861b-425036ebe3b8-service-ca\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.304802 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304691    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f869ae4c-2b69-48e5-861b-425036ebe3b8-console-config\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.304802 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304763    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f869ae4c-2b69-48e5-861b-425036ebe3b8-oauth-serving-cert\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.304924 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304824    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f869ae4c-2b69-48e5-861b-425036ebe3b8-console-serving-cert\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.304924 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304887    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt8n5\" (UniqueName: \"kubernetes.io/projected/f869ae4c-2b69-48e5-861b-425036ebe3b8-kube-api-access-rt8n5\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.304983 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304953    5663 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:08.304983 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304967    5663 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-serving-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:08.304983 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304976    5663 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-oauth-serving-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:08.305072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304986    5663 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59c1bd59-6c93-426a-9e81-a410a9cc62f4-tmp\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:08.305072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.304995    5663 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-trusted-ca-bundle\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:08.305072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.305005    5663 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-oauth-config\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:08.305072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.305014    5663 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59c1bd59-6c93-426a-9e81-a410a9cc62f4-console-config\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:08.305072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.305022    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhf7n\" (UniqueName: \"kubernetes.io/projected/59c1bd59-6c93-426a-9e81-a410a9cc62f4-kube-api-access-fhf7n\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:08.405838 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.405801    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f869ae4c-2b69-48e5-861b-425036ebe3b8-oauth-serving-cert\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.405838 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.405840    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f869ae4c-2b69-48e5-861b-425036ebe3b8-console-serving-cert\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.406039 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.405865    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt8n5\" (UniqueName: \"kubernetes.io/projected/f869ae4c-2b69-48e5-861b-425036ebe3b8-kube-api-access-rt8n5\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.406039 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.405939    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f869ae4c-2b69-48e5-861b-425036ebe3b8-tmp\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.406039 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.406000    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f869ae4c-2b69-48e5-861b-425036ebe3b8-console-oauth-config\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.406039 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.406034    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f869ae4c-2b69-48e5-861b-425036ebe3b8-trusted-ca-bundle\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.406230 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.406163    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f869ae4c-2b69-48e5-861b-425036ebe3b8-service-ca\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.406230 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.406211    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f869ae4c-2b69-48e5-861b-425036ebe3b8-console-config\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.406448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.406423    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f869ae4c-2b69-48e5-861b-425036ebe3b8-tmp\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.406697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.406673    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f869ae4c-2b69-48e5-861b-425036ebe3b8-oauth-serving-cert\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.406934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.406911    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f869ae4c-2b69-48e5-861b-425036ebe3b8-trusted-ca-bundle\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.406976 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.406957    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f869ae4c-2b69-48e5-861b-425036ebe3b8-service-ca\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.407008 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.406957    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f869ae4c-2b69-48e5-861b-425036ebe3b8-console-config\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.408338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.408318    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f869ae4c-2b69-48e5-861b-425036ebe3b8-console-oauth-config\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.408490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.408474    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f869ae4c-2b69-48e5-861b-425036ebe3b8-console-serving-cert\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.417847 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.417827    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt8n5\" (UniqueName: \"kubernetes.io/projected/f869ae4c-2b69-48e5-861b-425036ebe3b8-kube-api-access-rt8n5\") pod \"console-555fc688cb-25l7p\" (UID: \"f869ae4c-2b69-48e5-861b-425036ebe3b8\") " pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.452556 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.452519    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:08.725637 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.725604    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-555fc688cb-25l7p"]
Oct 31 12:29:08.728896 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:29:08.728870    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf869ae4c_2b69_48e5_861b_425036ebe3b8.slice/crio-5706348f374640a700ec1629b9f7c20d3cdcf1b44d8a55c664d06828f8600590 WatchSource:0}: Error finding container 5706348f374640a700ec1629b9f7c20d3cdcf1b44d8a55c664d06828f8600590: Status 404 returned error can't find the container with id 5706348f374640a700ec1629b9f7c20d3cdcf1b44d8a55c664d06828f8600590
Oct 31 12:29:08.787950 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.787906    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-555fc688cb-25l7p" event={"ID":"f869ae4c-2b69-48e5-861b-425036ebe3b8","Type":"ContainerStarted","Data":"13862bfc7fa06926238b86352f597ee2fac7366b4bd662d05f033a144e49914e"}
Oct 31 12:29:08.788410 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.787966    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-555fc688cb-25l7p" event={"ID":"f869ae4c-2b69-48e5-861b-425036ebe3b8","Type":"ContainerStarted","Data":"5706348f374640a700ec1629b9f7c20d3cdcf1b44d8a55c664d06828f8600590"}
Oct 31 12:29:08.789701 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.789671    5663 generic.go:356] "Generic (PLEG): container finished" podID="59c1bd59-6c93-426a-9e81-a410a9cc62f4" containerID="1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321" exitCode=0
Oct 31 12:29:08.789833 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.789708    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d5579bc6-rhzg5"
Oct 31 12:29:08.789833 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.789721    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d5579bc6-rhzg5" event={"ID":"59c1bd59-6c93-426a-9e81-a410a9cc62f4","Type":"ContainerDied","Data":"1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321"}
Oct 31 12:29:08.789833 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.789763    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d5579bc6-rhzg5" event={"ID":"59c1bd59-6c93-426a-9e81-a410a9cc62f4","Type":"ContainerDied","Data":"c5b8b9b880947cf81957233c5d94ee1156d415144bf00c0eb1eb667c7949e657"}
Oct 31 12:29:08.789833 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.789788    5663 scope.go:122] "RemoveContainer" containerID="1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321"
Oct 31 12:29:08.799596 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.799500    5663 scope.go:122] "RemoveContainer" containerID="1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321"
Oct 31 12:29:08.800034 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:29:08.800008    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321\": container with ID starting with 1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321 not found: ID does not exist" containerID="1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321"
Oct 31 12:29:08.800138 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.800040    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321"} err="failed to get container status \"1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321\": rpc error: code = NotFound desc = could not find container \"1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321\": container with ID starting with 1fcf27ef0c7dbc0b608f694f66227440262cb1ff8045ec313db397ee004aa321 not found: ID does not exist"
Oct 31 12:29:08.810682 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.810625    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-555fc688cb-25l7p" podStartSLOduration=0.810605486 podStartE2EDuration="810.605486ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:29:08.810152027 +0000 UTC m=+73.550578243" watchObservedRunningTime="2026-10-31 12:29:08.810605486 +0000 UTC m=+73.551031702"
Oct 31 12:29:08.825047 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.825019    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65d5579bc6-rhzg5"]
Oct 31 12:29:08.831295 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:08.831270    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65d5579bc6-rhzg5"]
Oct 31 12:29:09.436261 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:09.436219    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c1bd59-6c93-426a-9e81-a410a9cc62f4" path="/var/lib/kubelet/pods/59c1bd59-6c93-426a-9e81-a410a9cc62f4/volumes"
Oct 31 12:29:18.453084 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:18.453038    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:18.453084 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:18.453102    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:18.457850 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:18.457827    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:18.828539 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:18.828511    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-555fc688cb-25l7p"
Oct 31 12:29:18.877365 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:18.877324    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d674879f-l9xgc"]
Oct 31 12:29:22.474594 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:29:22.474560    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:29:43.896729 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:43.896646    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-69d674879f-l9xgc" podUID="a10ea48b-480f-4d01-8884-73286c7cb255" containerName="console" containerID="cri-o://1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14" gracePeriod=15
Oct 31 12:29:44.135143 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.135118    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:29:44.176946 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.176873    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-oauth-serving-cert\") pod \"a10ea48b-480f-4d01-8884-73286c7cb255\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") "
Oct 31 12:29:44.176946 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.176910    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-oauth-config\") pod \"a10ea48b-480f-4d01-8884-73286c7cb255\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") "
Oct 31 12:29:44.177142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.176957    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-trusted-ca-bundle\") pod \"a10ea48b-480f-4d01-8884-73286c7cb255\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") "
Oct 31 12:29:44.177142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.176977    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-console-config\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-console-config\") pod \"a10ea48b-480f-4d01-8884-73286c7cb255\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") "
Oct 31 12:29:44.177142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.177023    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-serving-cert\") pod \"a10ea48b-480f-4d01-8884-73286c7cb255\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") "
Oct 31 12:29:44.177142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.177073    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-service-ca\") pod \"a10ea48b-480f-4d01-8884-73286c7cb255\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") "
Oct 31 12:29:44.177142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.177093    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10ea48b-480f-4d01-8884-73286c7cb255-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10ea48b-480f-4d01-8884-73286c7cb255-tmp\") pod \"a10ea48b-480f-4d01-8884-73286c7cb255\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") "
Oct 31 12:29:44.177142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.177117    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10ea48b-480f-4d01-8884-73286c7cb255-kube-api-access-9ctqf\" (UniqueName: \"kubernetes.io/projected/a10ea48b-480f-4d01-8884-73286c7cb255-kube-api-access-9ctqf\") pod \"a10ea48b-480f-4d01-8884-73286c7cb255\" (UID: \"a10ea48b-480f-4d01-8884-73286c7cb255\") "
Oct 31 12:29:44.177475 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.177278    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-oauth-serving-cert" pod "a10ea48b-480f-4d01-8884-73286c7cb255" (UID: "a10ea48b-480f-4d01-8884-73286c7cb255"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:29:44.177475 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.177400    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-console-config" pod "a10ea48b-480f-4d01-8884-73286c7cb255" (UID: "a10ea48b-480f-4d01-8884-73286c7cb255"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:29:44.177578 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.177532    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10ea48b-480f-4d01-8884-73286c7cb255-tmp" pod "a10ea48b-480f-4d01-8884-73286c7cb255" (UID: "a10ea48b-480f-4d01-8884-73286c7cb255"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:29:44.177636 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.177563    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-trusted-ca-bundle" pod "a10ea48b-480f-4d01-8884-73286c7cb255" (UID: "a10ea48b-480f-4d01-8884-73286c7cb255"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:29:44.177710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.177685    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-service-ca" pod "a10ea48b-480f-4d01-8884-73286c7cb255" (UID: "a10ea48b-480f-4d01-8884-73286c7cb255"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:29:44.179217 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.179189    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-serving-cert" pod "a10ea48b-480f-4d01-8884-73286c7cb255" (UID: "a10ea48b-480f-4d01-8884-73286c7cb255"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:29:44.179217 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.179201    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-oauth-config" pod "a10ea48b-480f-4d01-8884-73286c7cb255" (UID: "a10ea48b-480f-4d01-8884-73286c7cb255"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:29:44.179217 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.179213    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10ea48b-480f-4d01-8884-73286c7cb255-kube-api-access-9ctqf" pod "a10ea48b-480f-4d01-8884-73286c7cb255" (UID: "a10ea48b-480f-4d01-8884-73286c7cb255"). InnerVolumeSpecName "kube-api-access-9ctqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:29:44.278183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.278156    5663 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-serving-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:44.278183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.278180    5663 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:44.278183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.278191    5663 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10ea48b-480f-4d01-8884-73286c7cb255-tmp\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:44.278432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.278200    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ctqf\" (UniqueName: \"kubernetes.io/projected/a10ea48b-480f-4d01-8884-73286c7cb255-kube-api-access-9ctqf\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:44.278432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.278209    5663 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-oauth-serving-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:44.278432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.278217    5663 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a10ea48b-480f-4d01-8884-73286c7cb255-console-oauth-config\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:44.278432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.278226    5663 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-trusted-ca-bundle\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:44.278432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.278235    5663 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a10ea48b-480f-4d01-8884-73286c7cb255-console-config\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:29:44.908033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.907996    5663 generic.go:356] "Generic (PLEG): container finished" podID="a10ea48b-480f-4d01-8884-73286c7cb255" containerID="1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14" exitCode=0
Oct 31 12:29:44.908479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.908044    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d674879f-l9xgc"
Oct 31 12:29:44.908479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.908084    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d674879f-l9xgc" event={"ID":"a10ea48b-480f-4d01-8884-73286c7cb255","Type":"ContainerDied","Data":"1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14"}
Oct 31 12:29:44.908479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.908127    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d674879f-l9xgc" event={"ID":"a10ea48b-480f-4d01-8884-73286c7cb255","Type":"ContainerDied","Data":"efe265b98f18c6b6bbe7f1f2a45de06ca75eaa2559209a33dd103be78eeebc7d"}
Oct 31 12:29:44.908479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.908145    5663 scope.go:122] "RemoveContainer" containerID="1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14"
Oct 31 12:29:44.916243 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.916227    5663 scope.go:122] "RemoveContainer" containerID="1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14"
Oct 31 12:29:44.916526 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:29:44.916507    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14\": container with ID starting with 1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14 not found: ID does not exist" containerID="1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14"
Oct 31 12:29:44.916600 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.916537    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14"} err="failed to get container status \"1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14\": rpc error: code = NotFound desc = could not find container \"1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14\": container with ID starting with 1f64760747a3da8c6647f083030b0cc3bd7b3d21ace6d5f6ae9655a749fa8e14 not found: ID does not exist"
Oct 31 12:29:44.928163 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.928139    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d674879f-l9xgc"]
Oct 31 12:29:44.931453 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:44.931426    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69d674879f-l9xgc"]
Oct 31 12:29:45.436521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:29:45.436486    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10ea48b-480f-4d01-8884-73286c7cb255" path="/var/lib/kubelet/pods/a10ea48b-480f-4d01-8884-73286c7cb255/volumes"
Oct 31 12:29:52.474026 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:29:52.473988    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:30:22.474089 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:30:22.474052    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:30:52.474728 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:30:52.474692    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:31:22.474060 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:31:22.474021    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:31:36.427960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.427929    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"]
Oct 31 12:31:36.428477 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.428213    5663 cpu_manager.go:414] "containerMap: removing container" podUID="a10ea48b-480f-4d01-8884-73286c7cb255" containerName="console"
Oct 31 12:31:36.428477 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.428227    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10ea48b-480f-4d01-8884-73286c7cb255" containerName="console"
Oct 31 12:31:36.428477 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.428284    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10ea48b-480f-4d01-8884-73286c7cb255" containerName="console"
Oct 31 12:31:36.431160 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.431143    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:31:36.439014 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.438995    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 12:31:36.439599 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.439580    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-mvxms\""
Oct 31 12:31:36.442711 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.442695    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 12:31:36.501211 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.501177    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2\" (UID: \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:31:36.501334 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.501295    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zz2x\" (UniqueName: \"kubernetes.io/projected/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-kube-api-access-5zz2x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2\" (UID: \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:31:36.501400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.501335    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2\" (UID: \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:31:36.511530 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.511461    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"]
Oct 31 12:31:36.602286 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.602257    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz2x\" (UniqueName: \"kubernetes.io/projected/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-kube-api-access-5zz2x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2\" (UID: \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:31:36.602456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.602296    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2\" (UID: \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:31:36.602456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.602320    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2\" (UID: \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:31:36.602743 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.602722    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2\" (UID: \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:31:36.602785 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.602736    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2\" (UID: \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:31:36.631748 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.631725    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz2x\" (UniqueName: \"kubernetes.io/projected/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-kube-api-access-5zz2x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2\" (UID: \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:31:36.664946 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.664917    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"]
Oct 31 12:31:36.668194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.668176    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.684019 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.683952    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 12:31:36.684019 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.683956    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mvht9\""
Oct 31 12:31:36.684019 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.683961    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 12:31:36.684019 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.684018    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 12:31:36.684209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.684028    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 12:31:36.684209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.684050    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 12:31:36.684209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.684054    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 12:31:36.696331 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.696305    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"]
Oct 31 12:31:36.746764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.746735    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:31:36.804001 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.803920    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c27230-6938-471e-9a6f-4655a4ed7cc4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.804136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.804070    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.804136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.804101    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.804259 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.804174    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpfvl\" (UniqueName: \"kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-kube-api-access-fpfvl\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.804259 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.804235    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.804396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.804281    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.804396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.804311    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.905489 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.905448    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.905489 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.905492    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.905729 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.905520    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpfvl\" (UniqueName: \"kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-kube-api-access-fpfvl\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.905729 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.905552    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.905729 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.905590    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.905729 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.905616    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.905729 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.905661    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c27230-6938-471e-9a6f-4655a4ed7cc4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.906156 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.906133    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.907940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.907910    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.908047 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.908001    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.908196 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.908180    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c27230-6938-471e-9a6f-4655a4ed7cc4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.908346 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.908329    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.920073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.920049    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpfvl\" (UniqueName: \"kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-kube-api-access-fpfvl\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.920476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.920454    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-nbb5j\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:36.978965 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:36.978881    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:37.027540 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:37.026707    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"]
Oct 31 12:31:37.031616 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:31:37.031583    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebcdd0a1_9028_409c_b66f_ef3e587c17ef.slice/crio-26cb8cba8f95e69c4dd0e797fb744886ddd54a4366dfb22f166624a33c25d07d WatchSource:0}: Error finding container 26cb8cba8f95e69c4dd0e797fb744886ddd54a4366dfb22f166624a33c25d07d: Status 404 returned error can't find the container with id 26cb8cba8f95e69c4dd0e797fb744886ddd54a4366dfb22f166624a33c25d07d
Oct 31 12:31:37.252424 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:37.252304    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"]
Oct 31 12:31:37.255599 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:31:37.255573    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c27230_6938_471e_9a6f_4655a4ed7cc4.slice/crio-38d5eeeff6981b42101f412ce7272d1874ae69ebdef7e27a5b9def1554cd26a6 WatchSource:0}: Error finding container 38d5eeeff6981b42101f412ce7272d1874ae69ebdef7e27a5b9def1554cd26a6: Status 404 returned error can't find the container with id 38d5eeeff6981b42101f412ce7272d1874ae69ebdef7e27a5b9def1554cd26a6
Oct 31 12:31:37.260598 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:37.260574    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j" event={"ID":"71c27230-6938-471e-9a6f-4655a4ed7cc4","Type":"ContainerStarted","Data":"38d5eeeff6981b42101f412ce7272d1874ae69ebdef7e27a5b9def1554cd26a6"}
Oct 31 12:31:37.261663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:37.261631    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2" event={"ID":"ebcdd0a1-9028-409c-b66f-ef3e587c17ef","Type":"ContainerStarted","Data":"26cb8cba8f95e69c4dd0e797fb744886ddd54a4366dfb22f166624a33c25d07d"}
Oct 31 12:31:41.940145 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:41.939736    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:31:41.940145 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:41.939798    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:31:42.282503 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:42.282421    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j" event={"ID":"71c27230-6938-471e-9a6f-4655a4ed7cc4","Type":"ContainerStarted","Data":"40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f"}
Oct 31 12:31:42.282661 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:42.282603    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:42.284267 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:42.284233    5663 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-nbb5j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:31:42.284409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:42.284283    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j" podUID="71c27230-6938-471e-9a6f-4655a4ed7cc4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:31:42.307203 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:42.307155    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j" podStartSLOduration=1.625950545 podStartE2EDuration="6.307142669s" totalImagesPullingTime="4.681192124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:31:42.305161393 +0000 UTC m=+227.045587612" watchObservedRunningTime="2026-10-31 12:31:42.307142669 +0000 UTC m=+227.047568884"
Oct 31 12:31:43.286373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:43.286316    5663 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-nbb5j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:31:43.286789 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:43.286405    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j" podUID="71c27230-6938-471e-9a6f-4655a4ed7cc4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:31:46.286813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:46.286780    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:31:46.297366 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:46.297324    5663 generic.go:356] "Generic (PLEG): container finished" podID="ebcdd0a1-9028-409c-b66f-ef3e587c17ef" containerID="3dfb07511daa8f93af9578aebd9c7ab1fee2f57d596fe6e5938a71f953d9791a" exitCode=0
Oct 31 12:31:46.297494 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:46.297388    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2" event={"ID":"ebcdd0a1-9028-409c-b66f-ef3e587c17ef","Type":"ContainerDied","Data":"3dfb07511daa8f93af9578aebd9c7ab1fee2f57d596fe6e5938a71f953d9791a"}
Oct 31 12:31:47.302895 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:47.302854    5663 generic.go:356] "Generic (PLEG): container finished" podID="ebcdd0a1-9028-409c-b66f-ef3e587c17ef" containerID="aa9b1696825285d2f6b910d681b6379ec275aac484652ac6cc22bd3a81f8c159" exitCode=0
Oct 31 12:31:47.303318 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:47.302918    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2" event={"ID":"ebcdd0a1-9028-409c-b66f-ef3e587c17ef","Type":"ContainerDied","Data":"aa9b1696825285d2f6b910d681b6379ec275aac484652ac6cc22bd3a81f8c159"}
Oct 31 12:31:52.474605 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:31:52.474566    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:31:58.348159 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:58.348123    5663 generic.go:356] "Generic (PLEG): container finished" podID="ebcdd0a1-9028-409c-b66f-ef3e587c17ef" containerID="5d4d8fb144fe4d5da354e1e7772a4024b0e1b4fd130a2f49c84cd87eb8171e92" exitCode=0
Oct 31 12:31:58.348633 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:58.348211    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2" event={"ID":"ebcdd0a1-9028-409c-b66f-ef3e587c17ef","Type":"ContainerDied","Data":"5d4d8fb144fe4d5da354e1e7772a4024b0e1b4fd130a2f49c84cd87eb8171e92"}
Oct 31 12:31:59.490198 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:59.490170    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:31:59.607150 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:59.607066    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-kube-api-access-5zz2x\" (UniqueName: \"kubernetes.io/projected/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-kube-api-access-5zz2x\") pod \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\" (UID: \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\") "
Oct 31 12:31:59.607150 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:59.607114    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-bundle\") pod \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\" (UID: \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\") "
Oct 31 12:31:59.607392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:59.607203    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-util\" (UniqueName: \"kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-util\") pod \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\" (UID: \"ebcdd0a1-9028-409c-b66f-ef3e587c17ef\") "
Oct 31 12:31:59.607806 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:59.607780    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-bundle" pod "ebcdd0a1-9028-409c-b66f-ef3e587c17ef" (UID: "ebcdd0a1-9028-409c-b66f-ef3e587c17ef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:31:59.609253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:59.609231    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-kube-api-access-5zz2x" pod "ebcdd0a1-9028-409c-b66f-ef3e587c17ef" (UID: "ebcdd0a1-9028-409c-b66f-ef3e587c17ef"). InnerVolumeSpecName "kube-api-access-5zz2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:31:59.612227 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:59.612204    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-util" pod "ebcdd0a1-9028-409c-b66f-ef3e587c17ef" (UID: "ebcdd0a1-9028-409c-b66f-ef3e587c17ef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:31:59.708675 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:59.708643    5663 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-util\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:31:59.708675 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:59.708673    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zz2x\" (UniqueName: \"kubernetes.io/projected/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-kube-api-access-5zz2x\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:31:59.708675 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:31:59.708684    5663 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebcdd0a1-9028-409c-b66f-ef3e587c17ef-bundle\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:32:00.359522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:00.359494    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2"
Oct 31 12:32:00.359688 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:00.359498    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mvl2" event={"ID":"ebcdd0a1-9028-409c-b66f-ef3e587c17ef","Type":"ContainerDied","Data":"26cb8cba8f95e69c4dd0e797fb744886ddd54a4366dfb22f166624a33c25d07d"}
Oct 31 12:32:00.359688 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:00.359617    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26cb8cba8f95e69c4dd0e797fb744886ddd54a4366dfb22f166624a33c25d07d"
Oct 31 12:32:22.473913 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:32:22.473872    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:32:35.246654 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.246571    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"]
Oct 31 12:32:35.247015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.246888    5663 cpu_manager.go:414] "containerMap: removing container" podUID="ebcdd0a1-9028-409c-b66f-ef3e587c17ef" containerName="pull"
Oct 31 12:32:35.247015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.246902    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebcdd0a1-9028-409c-b66f-ef3e587c17ef" containerName="pull"
Oct 31 12:32:35.247015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.246920    5663 cpu_manager.go:414] "containerMap: removing container" podUID="ebcdd0a1-9028-409c-b66f-ef3e587c17ef" containerName="util"
Oct 31 12:32:35.247015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.246925    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebcdd0a1-9028-409c-b66f-ef3e587c17ef" containerName="util"
Oct 31 12:32:35.247015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.246934    5663 cpu_manager.go:414] "containerMap: removing container" podUID="ebcdd0a1-9028-409c-b66f-ef3e587c17ef" containerName="extract"
Oct 31 12:32:35.247015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.246939    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebcdd0a1-9028-409c-b66f-ef3e587c17ef" containerName="extract"
Oct 31 12:32:35.247015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.246993    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebcdd0a1-9028-409c-b66f-ef3e587c17ef" containerName="extract"
Oct 31 12:32:35.248666 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.248637    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.289905 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.289877    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"]
Oct 31 12:32:35.334787 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.334753    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 12:32:35.335196 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.335175    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 12:32:35.335249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.335225    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 12:32:35.335293 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.335284    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 12:32:35.335458 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.335445    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-b82ch\""
Oct 31 12:32:35.335572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.335560    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 12:32:35.378077 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.378042    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-metrics-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.378227 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.378095    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-webhook-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.378227 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.378200    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-manifests\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.378300 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.378264    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.378300 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.378283    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-tmp\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.378397 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.378304    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4bb\" (UniqueName: \"kubernetes.io/projected/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-kube-api-access-gw4bb\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.478616 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.478590    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-metrics-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.478849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.478630    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-webhook-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.478849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.478651    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-manifests\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.478849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.478676    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.478849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.478813    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-tmp\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.479084 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.478854    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4bb\" (UniqueName: \"kubernetes.io/projected/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-kube-api-access-gw4bb\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.479139 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.479098    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-manifests\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.481169 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.481139    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.481336 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.481314    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-webhook-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.481401 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.481323    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-tmp\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.481493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.481476    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-metrics-cert\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.491025 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.491003    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4bb\" (UniqueName: \"kubernetes.io/projected/e0f58e20-1eb3-4ad9-b689-2e6298f1ba57-kube-api-access-gw4bb\") pod \"opendatahub-operator-controller-manager-5967ccc7d4-dg7s4\" (UID: \"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.560678 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.560657    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:35.856956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:35.856854    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"]
Oct 31 12:32:35.859886 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:32:35.859854    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f58e20_1eb3_4ad9_b689_2e6298f1ba57.slice/crio-3fa4b5c5dd712a2ed7f0f348cb695a6a774d578b66d863992eaf895a2b8a3290 WatchSource:0}: Error finding container 3fa4b5c5dd712a2ed7f0f348cb695a6a774d578b66d863992eaf895a2b8a3290: Status 404 returned error can't find the container with id 3fa4b5c5dd712a2ed7f0f348cb695a6a774d578b66d863992eaf895a2b8a3290
Oct 31 12:32:36.481574 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:36.481531    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4" event={"ID":"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57","Type":"ContainerStarted","Data":"3fa4b5c5dd712a2ed7f0f348cb695a6a774d578b66d863992eaf895a2b8a3290"}
Oct 31 12:32:39.494629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:39.494537    5663 generic.go:356] "Generic (PLEG): container finished" podID="e0f58e20-1eb3-4ad9-b689-2e6298f1ba57" containerID="d9a9bd37e1a9525ffc8f4ceb4cfbd9695f7f491c89cb90a5269a40d2350ac04d" exitCode=0
Oct 31 12:32:39.494629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:39.494618    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4" event={"ID":"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57","Type":"ContainerDied","Data":"d9a9bd37e1a9525ffc8f4ceb4cfbd9695f7f491c89cb90a5269a40d2350ac04d"}
Oct 31 12:32:40.502114 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:40.502071    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4" event={"ID":"e0f58e20-1eb3-4ad9-b689-2e6298f1ba57","Type":"ContainerStarted","Data":"ada078fd4ab295f43bc6c09a4f87f2c6694ea990172d7dabc112ee5ed72b0409"}
Oct 31 12:32:40.502666 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:40.502234    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:40.545620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:40.545569    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4" podStartSLOduration=1.853778113 podStartE2EDuration="5.545551049s" totalImagesPullingTime="3.265367108s" totalInitContainerRuntime="426.405828ms" isStatefulPod=false podCreationTimestamp="2026-10-31 12:32:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:32:40.543746629 +0000 UTC m=+285.284172878" watchObservedRunningTime="2026-10-31 12:32:40.545551049 +0000 UTC m=+285.285977262"
Oct 31 12:32:51.508609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:32:51.508578    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5967ccc7d4-dg7s4"
Oct 31 12:32:52.473997 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:32:52.473959    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:33:07.406046 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.406005    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"]
Oct 31 12:33:07.409451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.409423    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"
Oct 31 12:33:07.422672 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.422641    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"]
Oct 31 12:33:07.444245 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.444208    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzg9g\" (UniqueName: \"kubernetes.io/projected/631d337f-1016-400a-99df-436efb0db2e0-kube-api-access-rzg9g\") pod \"kube-auth-proxy-57cf7fb5f5-z927m\" (UID: \"631d337f-1016-400a-99df-436efb0db2e0\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"
Oct 31 12:33:07.444438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.444273    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/631d337f-1016-400a-99df-436efb0db2e0-tmp\") pod \"kube-auth-proxy-57cf7fb5f5-z927m\" (UID: \"631d337f-1016-400a-99df-436efb0db2e0\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"
Oct 31 12:33:07.444438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.444343    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631d337f-1016-400a-99df-436efb0db2e0-tls-certs\") pod \"kube-auth-proxy-57cf7fb5f5-z927m\" (UID: \"631d337f-1016-400a-99df-436efb0db2e0\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"
Oct 31 12:33:07.479108 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.479068    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 12:33:07.479108 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.479100    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 12:33:07.479313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.479195    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qprtw\""
Oct 31 12:33:07.544894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.544861    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631d337f-1016-400a-99df-436efb0db2e0-tls-certs\") pod \"kube-auth-proxy-57cf7fb5f5-z927m\" (UID: \"631d337f-1016-400a-99df-436efb0db2e0\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"
Oct 31 12:33:07.545080 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.544911    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzg9g\" (UniqueName: \"kubernetes.io/projected/631d337f-1016-400a-99df-436efb0db2e0-kube-api-access-rzg9g\") pod \"kube-auth-proxy-57cf7fb5f5-z927m\" (UID: \"631d337f-1016-400a-99df-436efb0db2e0\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"
Oct 31 12:33:07.545080 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.544986    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/631d337f-1016-400a-99df-436efb0db2e0-tmp\") pod \"kube-auth-proxy-57cf7fb5f5-z927m\" (UID: \"631d337f-1016-400a-99df-436efb0db2e0\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"
Oct 31 12:33:07.547818 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.547780    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631d337f-1016-400a-99df-436efb0db2e0-tls-certs\") pod \"kube-auth-proxy-57cf7fb5f5-z927m\" (UID: \"631d337f-1016-400a-99df-436efb0db2e0\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"
Oct 31 12:33:07.547972 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.547835    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/631d337f-1016-400a-99df-436efb0db2e0-tmp\") pod \"kube-auth-proxy-57cf7fb5f5-z927m\" (UID: \"631d337f-1016-400a-99df-436efb0db2e0\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"
Oct 31 12:33:07.569506 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.569478    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzg9g\" (UniqueName: \"kubernetes.io/projected/631d337f-1016-400a-99df-436efb0db2e0-kube-api-access-rzg9g\") pod \"kube-auth-proxy-57cf7fb5f5-z927m\" (UID: \"631d337f-1016-400a-99df-436efb0db2e0\") " pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"
Oct 31 12:33:07.724138 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:07.724056    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"
Oct 31 12:33:08.004201 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:08.004107    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m"]
Oct 31 12:33:08.007529 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:33:08.007486    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631d337f_1016_400a_99df_436efb0db2e0.slice/crio-f12da9c46d5742c3f84463c56512a122a0a0267c45b3bcadaaee6c58e6ea4a84 WatchSource:0}: Error finding container f12da9c46d5742c3f84463c56512a122a0a0267c45b3bcadaaee6c58e6ea4a84: Status 404 returned error can't find the container with id f12da9c46d5742c3f84463c56512a122a0a0267c45b3bcadaaee6c58e6ea4a84
Oct 31 12:33:08.009526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:08.009509    5663 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:33:08.612073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:08.612034    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m" event={"ID":"631d337f-1016-400a-99df-436efb0db2e0","Type":"ContainerStarted","Data":"f12da9c46d5742c3f84463c56512a122a0a0267c45b3bcadaaee6c58e6ea4a84"}
Oct 31 12:33:11.626550 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:11.626511    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m" event={"ID":"631d337f-1016-400a-99df-436efb0db2e0","Type":"ContainerStarted","Data":"7a3427af09e8d9c748b723e92eadf3dc01bbd68b4d72184021e8c1ed3601542a"}
Oct 31 12:33:11.654432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:11.654339    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57cf7fb5f5-z927m" podStartSLOduration=1.978887067 podStartE2EDuration="4.65432522s" totalImagesPullingTime="2.675438153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:11.653514763 +0000 UTC m=+316.393940980" watchObservedRunningTime="2026-10-31 12:33:11.65432522 +0000 UTC m=+316.394751505"
Oct 31 12:33:22.475419 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:33:22.475381    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:33:37.134951 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.134921    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"]
Oct 31 12:33:37.138027 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.138009    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"
Oct 31 12:33:37.144990 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.144973    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nszgg\""
Oct 31 12:33:37.150415 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.150391    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"]
Oct 31 12:33:37.150600 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.150587    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 12:33:37.150664 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.150645    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 12:33:37.150702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.150662    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 12:33:37.242280 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.242235    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"]
Oct 31 12:33:37.291580 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.291548    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31529c2c-0f46-41f2-b7fa-730e87df61e0-config-file\") pod \"limitador-limitador-74dfcfb894-tfsrf\" (UID: \"31529c2c-0f46-41f2-b7fa-730e87df61e0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"
Oct 31 12:33:37.291739 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.291586    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj8fs\" (UniqueName: \"kubernetes.io/projected/31529c2c-0f46-41f2-b7fa-730e87df61e0-kube-api-access-jj8fs\") pod \"limitador-limitador-74dfcfb894-tfsrf\" (UID: \"31529c2c-0f46-41f2-b7fa-730e87df61e0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"
Oct 31 12:33:37.392454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.392336    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31529c2c-0f46-41f2-b7fa-730e87df61e0-config-file\") pod \"limitador-limitador-74dfcfb894-tfsrf\" (UID: \"31529c2c-0f46-41f2-b7fa-730e87df61e0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"
Oct 31 12:33:37.392454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.392410    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj8fs\" (UniqueName: \"kubernetes.io/projected/31529c2c-0f46-41f2-b7fa-730e87df61e0-kube-api-access-jj8fs\") pod \"limitador-limitador-74dfcfb894-tfsrf\" (UID: \"31529c2c-0f46-41f2-b7fa-730e87df61e0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"
Oct 31 12:33:37.392992 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.392969    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31529c2c-0f46-41f2-b7fa-730e87df61e0-config-file\") pod \"limitador-limitador-74dfcfb894-tfsrf\" (UID: \"31529c2c-0f46-41f2-b7fa-730e87df61e0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"
Oct 31 12:33:37.405121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.405089    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj8fs\" (UniqueName: \"kubernetes.io/projected/31529c2c-0f46-41f2-b7fa-730e87df61e0-kube-api-access-jj8fs\") pod \"limitador-limitador-74dfcfb894-tfsrf\" (UID: \"31529c2c-0f46-41f2-b7fa-730e87df61e0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"
Oct 31 12:33:37.450823 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.450790    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"
Oct 31 12:33:37.723989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:37.723891    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"]
Oct 31 12:33:37.727092 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:33:37.727057    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31529c2c_0f46_41f2_b7fa_730e87df61e0.slice/crio-279e0b961717ba9c5347c9919d209328194a680dcf32e8ce8133658a101807a0 WatchSource:0}: Error finding container 279e0b961717ba9c5347c9919d209328194a680dcf32e8ce8133658a101807a0: Status 404 returned error can't find the container with id 279e0b961717ba9c5347c9919d209328194a680dcf32e8ce8133658a101807a0
Oct 31 12:33:38.723321 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:38.723278    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf" event={"ID":"31529c2c-0f46-41f2-b7fa-730e87df61e0","Type":"ContainerStarted","Data":"279e0b961717ba9c5347c9919d209328194a680dcf32e8ce8133658a101807a0"}
Oct 31 12:33:42.740329 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:42.740294    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf" event={"ID":"31529c2c-0f46-41f2-b7fa-730e87df61e0","Type":"ContainerStarted","Data":"d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe"}
Oct 31 12:33:42.740732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:42.740419    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"
Oct 31 12:33:42.758797 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:42.758753    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf" podStartSLOduration=1.595654194 podStartE2EDuration="5.758740062s" totalImagesPullingTime="4.163085868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:33:42.757646557 +0000 UTC m=+347.498072798" watchObservedRunningTime="2026-10-31 12:33:42.758740062 +0000 UTC m=+347.499166323"
Oct 31 12:33:51.722205 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:51.722167    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"]
Oct 31 12:33:51.722765 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:51.722389    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf" podUID="31529c2c-0f46-41f2-b7fa-730e87df61e0" containerName="limitador" containerID="cri-o://d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe" gracePeriod=30
Oct 31 12:33:51.723166 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:51.723146    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"
Oct 31 12:33:52.266440 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.266413    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"
Oct 31 12:33:52.426941 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.426905    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31529c2c-0f46-41f2-b7fa-730e87df61e0-kube-api-access-jj8fs\" (UniqueName: \"kubernetes.io/projected/31529c2c-0f46-41f2-b7fa-730e87df61e0-kube-api-access-jj8fs\") pod \"31529c2c-0f46-41f2-b7fa-730e87df61e0\" (UID: \"31529c2c-0f46-41f2-b7fa-730e87df61e0\") "
Oct 31 12:33:52.427105 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.426998    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31529c2c-0f46-41f2-b7fa-730e87df61e0-config-file\" (UniqueName: \"kubernetes.io/configmap/31529c2c-0f46-41f2-b7fa-730e87df61e0-config-file\") pod \"31529c2c-0f46-41f2-b7fa-730e87df61e0\" (UID: \"31529c2c-0f46-41f2-b7fa-730e87df61e0\") "
Oct 31 12:33:52.427400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.427379    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31529c2c-0f46-41f2-b7fa-730e87df61e0-config-file" pod "31529c2c-0f46-41f2-b7fa-730e87df61e0" (UID: "31529c2c-0f46-41f2-b7fa-730e87df61e0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:33:52.429191 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.429172    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31529c2c-0f46-41f2-b7fa-730e87df61e0-kube-api-access-jj8fs" pod "31529c2c-0f46-41f2-b7fa-730e87df61e0" (UID: "31529c2c-0f46-41f2-b7fa-730e87df61e0"). InnerVolumeSpecName "kube-api-access-jj8fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:33:52.475482 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:33:52.475448    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:33:52.527526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.527493    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj8fs\" (UniqueName: \"kubernetes.io/projected/31529c2c-0f46-41f2-b7fa-730e87df61e0-kube-api-access-jj8fs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:33:52.527526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.527524    5663 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31529c2c-0f46-41f2-b7fa-730e87df61e0-config-file\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:33:52.778463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.778375    5663 generic.go:356] "Generic (PLEG): container finished" podID="31529c2c-0f46-41f2-b7fa-730e87df61e0" containerID="d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe" exitCode=0
Oct 31 12:33:52.778463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.778413    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"
Oct 31 12:33:52.778463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.778422    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf" event={"ID":"31529c2c-0f46-41f2-b7fa-730e87df61e0","Type":"ContainerDied","Data":"d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe"}
Oct 31 12:33:52.779007 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.778475    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tfsrf" event={"ID":"31529c2c-0f46-41f2-b7fa-730e87df61e0","Type":"ContainerDied","Data":"279e0b961717ba9c5347c9919d209328194a680dcf32e8ce8133658a101807a0"}
Oct 31 12:33:52.779007 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.778505    5663 scope.go:122] "RemoveContainer" containerID="d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe"
Oct 31 12:33:52.786660 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.786636    5663 scope.go:122] "RemoveContainer" containerID="d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe"
Oct 31 12:33:52.787127 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:33:52.787101    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe\": container with ID starting with d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe not found: ID does not exist" containerID="d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe"
Oct 31 12:33:52.787191 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.787141    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe"} err="failed to get container status \"d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe\": rpc error: code = NotFound desc = could not find container \"d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe\": container with ID starting with d1b2e103fcd8beec22f7c45914e8248f40aa1432f1a351f70939f949cf43dafe not found: ID does not exist"
Oct 31 12:33:52.800248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.800223    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"]
Oct 31 12:33:52.808617 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:52.808594    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tfsrf"]
Oct 31 12:33:53.436254 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:53.436217    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31529c2c-0f46-41f2-b7fa-730e87df61e0" path="/var/lib/kubelet/pods/31529c2c-0f46-41f2-b7fa-730e87df61e0/volumes"
Oct 31 12:33:56.043345 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.043311    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"]
Oct 31 12:33:56.043730 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.043601    5663 cpu_manager.go:414] "containerMap: removing container" podUID="31529c2c-0f46-41f2-b7fa-730e87df61e0" containerName="limitador"
Oct 31 12:33:56.043730 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.043612    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31529c2c-0f46-41f2-b7fa-730e87df61e0" containerName="limitador"
Oct 31 12:33:56.043730 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.043680    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="31529c2c-0f46-41f2-b7fa-730e87df61e0" containerName="limitador"
Oct 31 12:33:56.048063 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.048033    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"
Oct 31 12:33:56.053062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.053037    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8d3eae84-8c5d-4fd2-b983-1d3e34d1507b-data\") pod \"postgres-7cdbf6dc44-wwt7t\" (UID: \"8d3eae84-8c5d-4fd2-b983-1d3e34d1507b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"
Oct 31 12:33:56.053193 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.053100    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5gxz\" (UniqueName: \"kubernetes.io/projected/8d3eae84-8c5d-4fd2-b983-1d3e34d1507b-kube-api-access-m5gxz\") pod \"postgres-7cdbf6dc44-wwt7t\" (UID: \"8d3eae84-8c5d-4fd2-b983-1d3e34d1507b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"
Oct 31 12:33:56.060325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.060299    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tdlnb\""
Oct 31 12:33:56.061327 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.061303    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 12:33:56.061635 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.061616    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 12:33:56.061713 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.061624    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 31 12:33:56.063054 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.063030    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"]
Oct 31 12:33:56.153904 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.153866    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8d3eae84-8c5d-4fd2-b983-1d3e34d1507b-data\") pod \"postgres-7cdbf6dc44-wwt7t\" (UID: \"8d3eae84-8c5d-4fd2-b983-1d3e34d1507b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"
Oct 31 12:33:56.154074 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.153918    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5gxz\" (UniqueName: \"kubernetes.io/projected/8d3eae84-8c5d-4fd2-b983-1d3e34d1507b-kube-api-access-m5gxz\") pod \"postgres-7cdbf6dc44-wwt7t\" (UID: \"8d3eae84-8c5d-4fd2-b983-1d3e34d1507b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"
Oct 31 12:33:56.154295 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.154276    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8d3eae84-8c5d-4fd2-b983-1d3e34d1507b-data\") pod \"postgres-7cdbf6dc44-wwt7t\" (UID: \"8d3eae84-8c5d-4fd2-b983-1d3e34d1507b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"
Oct 31 12:33:56.166176 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.166150    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5gxz\" (UniqueName: \"kubernetes.io/projected/8d3eae84-8c5d-4fd2-b983-1d3e34d1507b-kube-api-access-m5gxz\") pod \"postgres-7cdbf6dc44-wwt7t\" (UID: \"8d3eae84-8c5d-4fd2-b983-1d3e34d1507b\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"
Oct 31 12:33:56.371451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.371417    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"
Oct 31 12:33:56.645319 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.645243    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"]
Oct 31 12:33:56.648297 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:33:56.648264    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3eae84_8c5d_4fd2_b983_1d3e34d1507b.slice/crio-7585aa48b8fe398ea6421d709068e47eb18e620a23fee6829437b7935ea4102b WatchSource:0}: Error finding container 7585aa48b8fe398ea6421d709068e47eb18e620a23fee6829437b7935ea4102b: Status 404 returned error can't find the container with id 7585aa48b8fe398ea6421d709068e47eb18e620a23fee6829437b7935ea4102b
Oct 31 12:33:56.795478 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:33:56.795437    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t" event={"ID":"8d3eae84-8c5d-4fd2-b983-1d3e34d1507b","Type":"ContainerStarted","Data":"7585aa48b8fe398ea6421d709068e47eb18e620a23fee6829437b7935ea4102b"}
Oct 31 12:34:02.822765 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:02.822733    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t" event={"ID":"8d3eae84-8c5d-4fd2-b983-1d3e34d1507b","Type":"ContainerStarted","Data":"261fc271cd486f1dd0e17baa0f7d4246c784c192251d845e83f21f6d237c5d0b"}
Oct 31 12:34:02.823239 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:02.822872    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"
Oct 31 12:34:02.916800 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:02.916747    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t" podStartSLOduration=1.162044243 podStartE2EDuration="6.91673257s" totalImagesPullingTime="5.754688327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:34:02.916688628 +0000 UTC m=+367.657114856" watchObservedRunningTime="2026-10-31 12:34:02.91673257 +0000 UTC m=+367.657158787"
Oct 31 12:34:06.273463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.273428    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dwk2j"]
Oct 31 12:34:06.276585 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.276567    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"]
Oct 31 12:34:06.276715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.276695    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j"
Oct 31 12:34:06.279605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.279587    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"
Oct 31 12:34:06.305779 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.305751    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 12:34:06.305902 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.305805    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 12:34:06.312070 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.312048    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 31 12:34:06.312874 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.312858    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bnrvw\""
Oct 31 12:34:06.312949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.312874    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 31 12:34:06.312949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.312885    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-ssbf6\""
Oct 31 12:34:06.322448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.322409    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-wd2j4"]
Oct 31 12:34:06.326054 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.326032    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:06.327247 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.327226    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"]
Oct 31 12:34:06.330617 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.330594    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd96601d-214f-4e8f-9003-a78e49ba51a2-cert\") pod \"kserve-controller-manager-776cc66d47-dwk2j\" (UID: \"dd96601d-214f-4e8f-9003-a78e49ba51a2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j"
Oct 31 12:34:06.330716 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.330703    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772z8\" (UniqueName: \"kubernetes.io/projected/4fe387a8-f75e-4acb-973d-06f2bf25c8be-kube-api-access-772z8\") pod \"llmisvc-controller-manager-869d5dcc5d-5cqdf\" (UID: \"4fe387a8-f75e-4acb-973d-06f2bf25c8be\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"
Oct 31 12:34:06.330780 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.330752    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckqmz\" (UniqueName: \"kubernetes.io/projected/dd96601d-214f-4e8f-9003-a78e49ba51a2-kube-api-access-ckqmz\") pod \"kserve-controller-manager-776cc66d47-dwk2j\" (UID: \"dd96601d-214f-4e8f-9003-a78e49ba51a2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j"
Oct 31 12:34:06.330780 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.330777    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fe387a8-f75e-4acb-973d-06f2bf25c8be-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5cqdf\" (UID: \"4fe387a8-f75e-4acb-973d-06f2bf25c8be\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"
Oct 31 12:34:06.349962 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.349934    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-wd2j4"]
Oct 31 12:34:06.357498 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.357312    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dwk2j"]
Oct 31 12:34:06.402054 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.402030    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 31 12:34:06.402253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.402236    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dq26q\""
Oct 31 12:34:06.402307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.402240    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 31 12:34:06.431837 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.431801    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fe387a8-f75e-4acb-973d-06f2bf25c8be-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5cqdf\" (UID: \"4fe387a8-f75e-4acb-973d-06f2bf25c8be\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"
Oct 31 12:34:06.431987 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.431840    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac8c23-7b52-45f5-b50f-649c72ac1fd7-cert\") pod \"odh-model-controller-769875db56-wd2j4\" (UID: \"fbac8c23-7b52-45f5-b50f-649c72ac1fd7\") " pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:06.431987 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.431880    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd96601d-214f-4e8f-9003-a78e49ba51a2-cert\") pod \"kserve-controller-manager-776cc66d47-dwk2j\" (UID: \"dd96601d-214f-4e8f-9003-a78e49ba51a2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j"
Oct 31 12:34:06.431987 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.431971    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbac8c23-7b52-45f5-b50f-649c72ac1fd7-metrics-certs\") pod \"odh-model-controller-769875db56-wd2j4\" (UID: \"fbac8c23-7b52-45f5-b50f-649c72ac1fd7\") " pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:06.432165 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.432073    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-597sr\" (UniqueName: \"kubernetes.io/projected/fbac8c23-7b52-45f5-b50f-649c72ac1fd7-kube-api-access-597sr\") pod \"odh-model-controller-769875db56-wd2j4\" (UID: \"fbac8c23-7b52-45f5-b50f-649c72ac1fd7\") " pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:06.432165 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.432150    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772z8\" (UniqueName: \"kubernetes.io/projected/4fe387a8-f75e-4acb-973d-06f2bf25c8be-kube-api-access-772z8\") pod \"llmisvc-controller-manager-869d5dcc5d-5cqdf\" (UID: \"4fe387a8-f75e-4acb-973d-06f2bf25c8be\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"
Oct 31 12:34:06.432279 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.432192    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqmz\" (UniqueName: \"kubernetes.io/projected/dd96601d-214f-4e8f-9003-a78e49ba51a2-kube-api-access-ckqmz\") pod \"kserve-controller-manager-776cc66d47-dwk2j\" (UID: \"dd96601d-214f-4e8f-9003-a78e49ba51a2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j"
Oct 31 12:34:06.434296 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.434268    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fe387a8-f75e-4acb-973d-06f2bf25c8be-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5cqdf\" (UID: \"4fe387a8-f75e-4acb-973d-06f2bf25c8be\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"
Oct 31 12:34:06.434491 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.434471    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd96601d-214f-4e8f-9003-a78e49ba51a2-cert\") pod \"kserve-controller-manager-776cc66d47-dwk2j\" (UID: \"dd96601d-214f-4e8f-9003-a78e49ba51a2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j"
Oct 31 12:34:06.461442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.461416    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-772z8\" (UniqueName: \"kubernetes.io/projected/4fe387a8-f75e-4acb-973d-06f2bf25c8be-kube-api-access-772z8\") pod \"llmisvc-controller-manager-869d5dcc5d-5cqdf\" (UID: \"4fe387a8-f75e-4acb-973d-06f2bf25c8be\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"
Oct 31 12:34:06.461442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.461428    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckqmz\" (UniqueName: \"kubernetes.io/projected/dd96601d-214f-4e8f-9003-a78e49ba51a2-kube-api-access-ckqmz\") pod \"kserve-controller-manager-776cc66d47-dwk2j\" (UID: \"dd96601d-214f-4e8f-9003-a78e49ba51a2\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j"
Oct 31 12:34:06.533290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.533218    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac8c23-7b52-45f5-b50f-649c72ac1fd7-cert\") pod \"odh-model-controller-769875db56-wd2j4\" (UID: \"fbac8c23-7b52-45f5-b50f-649c72ac1fd7\") " pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:06.533290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.533273    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbac8c23-7b52-45f5-b50f-649c72ac1fd7-metrics-certs\") pod \"odh-model-controller-769875db56-wd2j4\" (UID: \"fbac8c23-7b52-45f5-b50f-649c72ac1fd7\") " pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:06.533526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.533301    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-597sr\" (UniqueName: \"kubernetes.io/projected/fbac8c23-7b52-45f5-b50f-649c72ac1fd7-kube-api-access-597sr\") pod \"odh-model-controller-769875db56-wd2j4\" (UID: \"fbac8c23-7b52-45f5-b50f-649c72ac1fd7\") " pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:06.535678 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.535651    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbac8c23-7b52-45f5-b50f-649c72ac1fd7-metrics-certs\") pod \"odh-model-controller-769875db56-wd2j4\" (UID: \"fbac8c23-7b52-45f5-b50f-649c72ac1fd7\") " pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:06.535786 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.535720    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbac8c23-7b52-45f5-b50f-649c72ac1fd7-cert\") pod \"odh-model-controller-769875db56-wd2j4\" (UID: \"fbac8c23-7b52-45f5-b50f-649c72ac1fd7\") " pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:06.581723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.581692    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-597sr\" (UniqueName: \"kubernetes.io/projected/fbac8c23-7b52-45f5-b50f-649c72ac1fd7-kube-api-access-597sr\") pod \"odh-model-controller-769875db56-wd2j4\" (UID: \"fbac8c23-7b52-45f5-b50f-649c72ac1fd7\") " pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:06.590565 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.590543    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j"
Oct 31 12:34:06.595254 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.595236    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"]
Oct 31 12:34:06.595819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.595802    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"
Oct 31 12:34:06.598939 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.598924    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.619298 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.619224    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"]
Oct 31 12:34:06.636695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.634699    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.636695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.634742    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/081a90a7-3aa0-47c6-b528-c103b3a8dc62-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.636695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.634762    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.636695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.634778    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.636695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.634815    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn84g\" (UniqueName: \"kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-kube-api-access-kn84g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.636695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.634844    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.636695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.634864    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.641472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.641447    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:06.736082 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.736054    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.736253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.736099    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/081a90a7-3aa0-47c6-b528-c103b3a8dc62-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.736253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.736120    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.736253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.736140    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.736253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.736197    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn84g\" (UniqueName: \"kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-kube-api-access-kn84g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.736253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.736241    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.736528 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.736266    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.736972 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.736947    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.738508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.738487    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.738616 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.738572    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.738754 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.738734    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/081a90a7-3aa0-47c6-b528-c103b3a8dc62-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.738808 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.738791    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.755420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.755399    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.755725 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.755708    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn84g\" (UniqueName: \"kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-kube-api-access-kn84g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4ggxs\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.877472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.877442    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dwk2j"]
Oct 31 12:34:06.880393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.880365    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"]
Oct 31 12:34:06.881244 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:34:06.881215    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd96601d_214f_4e8f_9003_a78e49ba51a2.slice/crio-8c27b96adba0972dfd446479c85c3b6993f615c64f1fff49f79099bc86e8a88f WatchSource:0}: Error finding container 8c27b96adba0972dfd446479c85c3b6993f615c64f1fff49f79099bc86e8a88f: Status 404 returned error can't find the container with id 8c27b96adba0972dfd446479c85c3b6993f615c64f1fff49f79099bc86e8a88f
Oct 31 12:34:06.883075 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:34:06.883051    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4fe387a8_f75e_4acb_973d_06f2bf25c8be.slice/crio-9bfd00ba9cfa6bbfca6407680d5ce0f3670f01f2cef3ec46637893decc3a8ea8 WatchSource:0}: Error finding container 9bfd00ba9cfa6bbfca6407680d5ce0f3670f01f2cef3ec46637893decc3a8ea8: Status 404 returned error can't find the container with id 9bfd00ba9cfa6bbfca6407680d5ce0f3670f01f2cef3ec46637893decc3a8ea8
Oct 31 12:34:06.911768 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.911745    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:06.931574 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:06.931542    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-wd2j4"]
Oct 31 12:34:06.934664 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:34:06.934638    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbac8c23_7b52_45f5_b50f_649c72ac1fd7.slice/crio-3125a55516c0c5ab6e9fc00adbf0dde9a517d2a432662e892de023a3678e03eb WatchSource:0}: Error finding container 3125a55516c0c5ab6e9fc00adbf0dde9a517d2a432662e892de023a3678e03eb: Status 404 returned error can't find the container with id 3125a55516c0c5ab6e9fc00adbf0dde9a517d2a432662e892de023a3678e03eb
Oct 31 12:34:07.222034 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:34:07.221999    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081a90a7_3aa0_47c6_b528_c103b3a8dc62.slice/crio-2c35a067e311db3a2d2d8e3061cb0a02042208251fd1c71fc742ef622ac7a705 WatchSource:0}: Error finding container 2c35a067e311db3a2d2d8e3061cb0a02042208251fd1c71fc742ef622ac7a705: Status 404 returned error can't find the container with id 2c35a067e311db3a2d2d8e3061cb0a02042208251fd1c71fc742ef622ac7a705
Oct 31 12:34:07.224138 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:07.224106    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:34:07.224209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:07.224169    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:34:07.234890 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:07.234870    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"]
Oct 31 12:34:07.844989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:07.844949    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-wd2j4" event={"ID":"fbac8c23-7b52-45f5-b50f-649c72ac1fd7","Type":"ContainerStarted","Data":"3125a55516c0c5ab6e9fc00adbf0dde9a517d2a432662e892de023a3678e03eb"}
Oct 31 12:34:07.846762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:07.846734    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf" event={"ID":"4fe387a8-f75e-4acb-973d-06f2bf25c8be","Type":"ContainerStarted","Data":"9bfd00ba9cfa6bbfca6407680d5ce0f3670f01f2cef3ec46637893decc3a8ea8"}
Oct 31 12:34:07.848176 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:07.848154    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j" event={"ID":"dd96601d-214f-4e8f-9003-a78e49ba51a2","Type":"ContainerStarted","Data":"8c27b96adba0972dfd446479c85c3b6993f615c64f1fff49f79099bc86e8a88f"}
Oct 31 12:34:07.850493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:07.850413    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs" event={"ID":"081a90a7-3aa0-47c6-b528-c103b3a8dc62","Type":"ContainerStarted","Data":"38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a"}
Oct 31 12:34:07.850594 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:07.850509    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs" event={"ID":"081a90a7-3aa0-47c6-b528-c103b3a8dc62","Type":"ContainerStarted","Data":"2c35a067e311db3a2d2d8e3061cb0a02042208251fd1c71fc742ef622ac7a705"}
Oct 31 12:34:07.850594 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:07.850549    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:07.886363 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:07.886300    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs" podStartSLOduration=1.886281138 podStartE2EDuration="1.886281138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:34:07.885268519 +0000 UTC m=+372.625694737" watchObservedRunningTime="2026-10-31 12:34:07.886281138 +0000 UTC m=+372.626707356"
Oct 31 12:34:08.858220 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:08.858187    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 12:34:08.866509 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:08.866482    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wwt7t"
Oct 31 12:34:08.948630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:08.948597    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"]
Oct 31 12:34:08.948854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:08.948775    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j" podUID="71c27230-6938-471e-9a6f-4655a4ed7cc4" containerName="discovery" containerID="cri-o://40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f" gracePeriod=30
Oct 31 12:34:09.237419 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.237152    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:34:09.263034 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.263005    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-ca-configmap\") pod \"71c27230-6938-471e-9a6f-4655a4ed7cc4\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") "
Oct 31 12:34:09.263193 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.263048    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-token\" (UniqueName: \"kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-token\") pod \"71c27230-6938-471e-9a6f-4655a4ed7cc4\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") "
Oct 31 12:34:09.263193 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.263097    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-kubeconfig\") pod \"71c27230-6938-471e-9a6f-4655a4ed7cc4\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") "
Oct 31 12:34:09.263193 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.263129    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-dns-cert\") pod \"71c27230-6938-471e-9a6f-4655a4ed7cc4\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") "
Oct 31 12:34:09.263193 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.263167    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-kube-api-access-fpfvl\" (UniqueName: \"kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-kube-api-access-fpfvl\") pod \"71c27230-6938-471e-9a6f-4655a4ed7cc4\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") "
Oct 31 12:34:09.263448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.263214    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c27230-6938-471e-9a6f-4655a4ed7cc4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c27230-6938-471e-9a6f-4655a4ed7cc4-local-certs\") pod \"71c27230-6938-471e-9a6f-4655a4ed7cc4\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") "
Oct 31 12:34:09.263448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.263253    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-cacerts\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-cacerts\") pod \"71c27230-6938-471e-9a6f-4655a4ed7cc4\" (UID: \"71c27230-6938-471e-9a6f-4655a4ed7cc4\") "
Oct 31 12:34:09.263576 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.263448    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-ca-configmap" pod "71c27230-6938-471e-9a6f-4655a4ed7cc4" (UID: "71c27230-6938-471e-9a6f-4655a4ed7cc4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:34:09.263576 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.263557    5663 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-ca-configmap\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:34:09.266014 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.265984    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-kube-api-access-fpfvl" pod "71c27230-6938-471e-9a6f-4655a4ed7cc4" (UID: "71c27230-6938-471e-9a6f-4655a4ed7cc4"). InnerVolumeSpecName "kube-api-access-fpfvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:34:09.266293 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.266268    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-cacerts" pod "71c27230-6938-471e-9a6f-4655a4ed7cc4" (UID: "71c27230-6938-471e-9a6f-4655a4ed7cc4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:34:09.266401 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.266306    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-token" pod "71c27230-6938-471e-9a6f-4655a4ed7cc4" (UID: "71c27230-6938-471e-9a6f-4655a4ed7cc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:34:09.266401 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.266374    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-kubeconfig" pod "71c27230-6938-471e-9a6f-4655a4ed7cc4" (UID: "71c27230-6938-471e-9a6f-4655a4ed7cc4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:34:09.266523 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.266486    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c27230-6938-471e-9a6f-4655a4ed7cc4-local-certs" pod "71c27230-6938-471e-9a6f-4655a4ed7cc4" (UID: "71c27230-6938-471e-9a6f-4655a4ed7cc4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:34:09.266563 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.266528    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-dns-cert" pod "71c27230-6938-471e-9a6f-4655a4ed7cc4" (UID: "71c27230-6938-471e-9a6f-4655a4ed7cc4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:34:09.364314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.364158    5663 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-csr-dns-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:34:09.364314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.364207    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpfvl\" (UniqueName: \"kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-kube-api-access-fpfvl\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:34:09.364314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.364225    5663 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c27230-6938-471e-9a6f-4655a4ed7cc4-local-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:34:09.364314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.364248    5663 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-cacerts\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:34:09.364314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.364261    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:34:09.364314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.364273    5663 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/71c27230-6938-471e-9a6f-4655a4ed7cc4-istio-kubeconfig\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:34:09.864388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.863829    5663 generic.go:356] "Generic (PLEG): container finished" podID="71c27230-6938-471e-9a6f-4655a4ed7cc4" containerID="40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f" exitCode=0
Oct 31 12:34:09.865264 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.864419    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j" event={"ID":"71c27230-6938-471e-9a6f-4655a4ed7cc4","Type":"ContainerDied","Data":"40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f"}
Oct 31 12:34:09.865264 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.864475    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"
Oct 31 12:34:09.865264 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.864489    5663 scope.go:122] "RemoveContainer" containerID="40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f"
Oct 31 12:34:09.865264 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.864477    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j" event={"ID":"71c27230-6938-471e-9a6f-4655a4ed7cc4","Type":"ContainerDied","Data":"38d5eeeff6981b42101f412ce7272d1874ae69ebdef7e27a5b9def1554cd26a6"}
Oct 31 12:34:09.878871 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.878773    5663 scope.go:122] "RemoveContainer" containerID="40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f"
Oct 31 12:34:09.879157 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:34:09.879080    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f\": container with ID starting with 40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f not found: ID does not exist" containerID="40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f"
Oct 31 12:34:09.879157 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.879117    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f"} err="failed to get container status \"40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f\": rpc error: code = NotFound desc = could not find container \"40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f\": container with ID starting with 40b2e821c87d2cd928fe9c69aa526298a4e3c9feb02be6c91eb67bd3f8619a8f not found: ID does not exist"
Oct 31 12:34:09.902892 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.902863    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"]
Oct 31 12:34:09.912085 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:09.912045    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nbb5j"]
Oct 31 12:34:10.872582 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:10.872560    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf" event={"ID":"4fe387a8-f75e-4acb-973d-06f2bf25c8be","Type":"ContainerStarted","Data":"56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357"}
Oct 31 12:34:10.872916 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:10.872731    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"
Oct 31 12:34:10.901768 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:10.901718    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf" podStartSLOduration=1.783018625 podStartE2EDuration="4.901701141s" totalImagesPullingTime="3.118682516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:34:10.899935949 +0000 UTC m=+375.640362165" watchObservedRunningTime="2026-10-31 12:34:10.901701141 +0000 UTC m=+375.642127358"
Oct 31 12:34:11.436937 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:11.436905    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c27230-6938-471e-9a6f-4655a4ed7cc4" path="/var/lib/kubelet/pods/71c27230-6938-471e-9a6f-4655a4ed7cc4/volumes"
Oct 31 12:34:11.877259 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:11.877224    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j" event={"ID":"dd96601d-214f-4e8f-9003-a78e49ba51a2","Type":"ContainerStarted","Data":"72fc50aba2adbc20b1f36adbb62724a6f10ba739ed39b5c25e4c5c20ff0ebb20"}
Oct 31 12:34:11.877707 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:11.877283    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j"
Oct 31 12:34:11.878695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:11.878666    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-wd2j4" event={"ID":"fbac8c23-7b52-45f5-b50f-649c72ac1fd7","Type":"ContainerStarted","Data":"a9cce8632dd64dda0096e7b341ec8054d342157c15441c51b246cd93848b2a6c"}
Oct 31 12:34:11.908801 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:11.908751    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j" podStartSLOduration=1.93716924 podStartE2EDuration="5.908736684s" totalImagesPullingTime="3.971567444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:34:11.907050216 +0000 UTC m=+376.647476431" watchObservedRunningTime="2026-10-31 12:34:11.908736684 +0000 UTC m=+376.649162900"
Oct 31 12:34:11.930590 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:11.930544    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-wd2j4" podStartSLOduration=2.003498753 podStartE2EDuration="5.930529476s" totalImagesPullingTime="3.927030723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:34:11.928812449 +0000 UTC m=+376.669238667" watchObservedRunningTime="2026-10-31 12:34:11.930529476 +0000 UTC m=+376.670955691"
Oct 31 12:34:12.883170 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:12.883143    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:22.474845 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:34:22.474806    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:34:22.699481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.699448    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"]
Oct 31 12:34:22.699788 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.699775    5663 cpu_manager.go:414] "containerMap: removing container" podUID="71c27230-6938-471e-9a6f-4655a4ed7cc4" containerName="discovery"
Oct 31 12:34:22.699851 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.699788    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c27230-6938-471e-9a6f-4655a4ed7cc4" containerName="discovery"
Oct 31 12:34:22.699851 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.699845    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c27230-6938-471e-9a6f-4655a4ed7cc4" containerName="discovery"
Oct 31 12:34:22.710934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.710906    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:22.716041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.715827    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 31 12:34:22.716041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.715895    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 31 12:34:22.716041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.715980    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-5k9j4\""
Oct 31 12:34:22.716041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.716031    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 31 12:34:22.718126 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.716527    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"]
Oct 31 12:34:22.792328 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.792251    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/fff0b729-f78b-4bd7-ad0a-e66d5b14243d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wfkln\" (UID: \"fff0b729-f78b-4bd7-ad0a-e66d5b14243d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:22.792328 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.792289    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5ccb\" (UniqueName: \"kubernetes.io/projected/fff0b729-f78b-4bd7-ad0a-e66d5b14243d-kube-api-access-r5ccb\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wfkln\" (UID: \"fff0b729-f78b-4bd7-ad0a-e66d5b14243d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:22.792529 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.792425    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fff0b729-f78b-4bd7-ad0a-e66d5b14243d-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wfkln\" (UID: \"fff0b729-f78b-4bd7-ad0a-e66d5b14243d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:22.792529 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.792473    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fff0b729-f78b-4bd7-ad0a-e66d5b14243d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wfkln\" (UID: \"fff0b729-f78b-4bd7-ad0a-e66d5b14243d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:22.893391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.893338    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fff0b729-f78b-4bd7-ad0a-e66d5b14243d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wfkln\" (UID: \"fff0b729-f78b-4bd7-ad0a-e66d5b14243d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:22.893554 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.893408    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/fff0b729-f78b-4bd7-ad0a-e66d5b14243d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wfkln\" (UID: \"fff0b729-f78b-4bd7-ad0a-e66d5b14243d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:22.893554 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.893435    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ccb\" (UniqueName: \"kubernetes.io/projected/fff0b729-f78b-4bd7-ad0a-e66d5b14243d-kube-api-access-r5ccb\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wfkln\" (UID: \"fff0b729-f78b-4bd7-ad0a-e66d5b14243d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:22.893554 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.893478    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fff0b729-f78b-4bd7-ad0a-e66d5b14243d-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wfkln\" (UID: \"fff0b729-f78b-4bd7-ad0a-e66d5b14243d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:22.894312 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.894285    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fff0b729-f78b-4bd7-ad0a-e66d5b14243d-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wfkln\" (UID: \"fff0b729-f78b-4bd7-ad0a-e66d5b14243d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:22.894551 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.894532    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/fff0b729-f78b-4bd7-ad0a-e66d5b14243d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wfkln\" (UID: \"fff0b729-f78b-4bd7-ad0a-e66d5b14243d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:22.895750 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.895731    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fff0b729-f78b-4bd7-ad0a-e66d5b14243d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wfkln\" (UID: \"fff0b729-f78b-4bd7-ad0a-e66d5b14243d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:22.905869 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:22.905845    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5ccb\" (UniqueName: \"kubernetes.io/projected/fff0b729-f78b-4bd7-ad0a-e66d5b14243d-kube-api-access-r5ccb\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wfkln\" (UID: \"fff0b729-f78b-4bd7-ad0a-e66d5b14243d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:23.026394 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:23.026332    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:23.302766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:23.302737    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"]
Oct 31 12:34:23.305449 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:34:23.305412    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff0b729_f78b_4bd7_ad0a_e66d5b14243d.slice/crio-f7e8006ffc95ecd3a95b4b0e58d68c2d2b74fcd93d6bfa19bbdc931047b6bfbf WatchSource:0}: Error finding container f7e8006ffc95ecd3a95b4b0e58d68c2d2b74fcd93d6bfa19bbdc931047b6bfbf: Status 404 returned error can't find the container with id f7e8006ffc95ecd3a95b4b0e58d68c2d2b74fcd93d6bfa19bbdc931047b6bfbf
Oct 31 12:34:23.891069 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:23.891038    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-wd2j4"
Oct 31 12:34:23.931893 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:23.931849    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln" event={"ID":"fff0b729-f78b-4bd7-ad0a-e66d5b14243d","Type":"ContainerStarted","Data":"f7e8006ffc95ecd3a95b4b0e58d68c2d2b74fcd93d6bfa19bbdc931047b6bfbf"}
Oct 31 12:34:41.881758 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:41.881727    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"
Oct 31 12:34:42.888987 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:42.888945    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-dwk2j"
Oct 31 12:34:45.020775 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:45.020689    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln" event={"ID":"fff0b729-f78b-4bd7-ad0a-e66d5b14243d","Type":"ContainerStarted","Data":"02814cfbe4867d0bfe9a0465f8a7e831cbcbf7524d2c5d53fbcd866ac2ed23a8"}
Oct 31 12:34:45.020775 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:45.020770    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:45.042274 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:45.042224    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln" podStartSLOduration=1.647265056 podStartE2EDuration="23.042210034s" totalImagesPullingTime="21.394944978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:34:45.040790656 +0000 UTC m=+409.781216872" watchObservedRunningTime="2026-10-31 12:34:45.042210034 +0000 UTC m=+409.782636250"
Oct 31 12:34:52.229752 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.229717    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"]
Oct 31 12:34:52.232071 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.232047    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.240177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.239951    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 12:34:52.240177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.240081    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 12:34:52.240177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.240160    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6jkzx\""
Oct 31 12:34:52.249671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.249647    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"]
Oct 31 12:34:52.256697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.256676    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-metrics-certs\") pod \"maas-api-6d669c7fcd-bg7ck\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.256826 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.256746    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da963526-ab82-4672-84dc-21c68b7de535-openshift-service-ca\") pod \"maas-api-6d669c7fcd-bg7ck\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.256826 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.256801    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qtjn\" (UniqueName: \"kubernetes.io/projected/da963526-ab82-4672-84dc-21c68b7de535-kube-api-access-2qtjn\") pod \"maas-api-6d669c7fcd-bg7ck\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.256921 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.256906    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-maas-api-tls\") pod \"maas-api-6d669c7fcd-bg7ck\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.357778 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.357743    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-maas-api-tls\") pod \"maas-api-6d669c7fcd-bg7ck\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.357950 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.357787    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-metrics-certs\") pod \"maas-api-6d669c7fcd-bg7ck\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.357950 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.357840    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da963526-ab82-4672-84dc-21c68b7de535-openshift-service-ca\") pod \"maas-api-6d669c7fcd-bg7ck\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.357950 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.357857    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qtjn\" (UniqueName: \"kubernetes.io/projected/da963526-ab82-4672-84dc-21c68b7de535-kube-api-access-2qtjn\") pod \"maas-api-6d669c7fcd-bg7ck\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.358624 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.358601    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da963526-ab82-4672-84dc-21c68b7de535-openshift-service-ca\") pod \"maas-api-6d669c7fcd-bg7ck\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.360245 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.360218    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-maas-api-tls\") pod \"maas-api-6d669c7fcd-bg7ck\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.360380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.360258    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-metrics-certs\") pod \"maas-api-6d669c7fcd-bg7ck\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.373325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.373305    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qtjn\" (UniqueName: \"kubernetes.io/projected/da963526-ab82-4672-84dc-21c68b7de535-kube-api-access-2qtjn\") pod \"maas-api-6d669c7fcd-bg7ck\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.475146 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:34:52.475106    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:34:52.545583 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.545493    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:52.830823 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:52.830784    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"]
Oct 31 12:34:52.834719 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:34:52.834688    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda963526_ab82_4672_84dc_21c68b7de535.slice/crio-252e252eff26c219aae16cceda03c0317003609f06bd791703c375a4f34ba62f WatchSource:0}: Error finding container 252e252eff26c219aae16cceda03c0317003609f06bd791703c375a4f34ba62f: Status 404 returned error can't find the container with id 252e252eff26c219aae16cceda03c0317003609f06bd791703c375a4f34ba62f
Oct 31 12:34:53.053919 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:53.053877    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck" event={"ID":"da963526-ab82-4672-84dc-21c68b7de535","Type":"ContainerStarted","Data":"252e252eff26c219aae16cceda03c0317003609f06bd791703c375a4f34ba62f"}
Oct 31 12:34:56.030876 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:56.030809    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wfkln"
Oct 31 12:34:56.072393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:56.072341    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck" event={"ID":"da963526-ab82-4672-84dc-21c68b7de535","Type":"ContainerStarted","Data":"b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff"}
Oct 31 12:34:56.072569 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:56.072481    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:34:56.092662 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:34:56.092616    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck" podStartSLOduration=1.25263421 podStartE2EDuration="4.092602232s" totalImagesPullingTime="2.839968022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:34:56.091605665 +0000 UTC m=+420.832031882" watchObservedRunningTime="2026-10-31 12:34:56.092602232 +0000 UTC m=+420.833028447"
Oct 31 12:35:02.081832 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:02.081801    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:35:06.771079 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:06.771045    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"]
Oct 31 12:35:06.774431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:06.774411    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"
Oct 31 12:35:06.785502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:06.785472    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"]
Oct 31 12:35:06.887791 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:06.887756    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19431d95-2890-4c4c-b7b4-d7adb4fd51d5-cert\") pod \"llmisvc-controller-manager-6466c554d8-mt6xx\" (UID: \"19431d95-2890-4c4c-b7b4-d7adb4fd51d5\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"
Oct 31 12:35:06.887968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:06.887797    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hh6g\" (UniqueName: \"kubernetes.io/projected/19431d95-2890-4c4c-b7b4-d7adb4fd51d5-kube-api-access-5hh6g\") pod \"llmisvc-controller-manager-6466c554d8-mt6xx\" (UID: \"19431d95-2890-4c4c-b7b4-d7adb4fd51d5\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"
Oct 31 12:35:06.988890 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:06.988849    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19431d95-2890-4c4c-b7b4-d7adb4fd51d5-cert\") pod \"llmisvc-controller-manager-6466c554d8-mt6xx\" (UID: \"19431d95-2890-4c4c-b7b4-d7adb4fd51d5\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"
Oct 31 12:35:06.988890 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:06.988890    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hh6g\" (UniqueName: \"kubernetes.io/projected/19431d95-2890-4c4c-b7b4-d7adb4fd51d5-kube-api-access-5hh6g\") pod \"llmisvc-controller-manager-6466c554d8-mt6xx\" (UID: \"19431d95-2890-4c4c-b7b4-d7adb4fd51d5\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"
Oct 31 12:35:06.991439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:06.991412    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19431d95-2890-4c4c-b7b4-d7adb4fd51d5-cert\") pod \"llmisvc-controller-manager-6466c554d8-mt6xx\" (UID: \"19431d95-2890-4c4c-b7b4-d7adb4fd51d5\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"
Oct 31 12:35:07.001465 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:07.001436    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hh6g\" (UniqueName: \"kubernetes.io/projected/19431d95-2890-4c4c-b7b4-d7adb4fd51d5-kube-api-access-5hh6g\") pod \"llmisvc-controller-manager-6466c554d8-mt6xx\" (UID: \"19431d95-2890-4c4c-b7b4-d7adb4fd51d5\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"
Oct 31 12:35:07.086861 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:07.086829    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"
Oct 31 12:35:07.367075 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:07.367000    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"]
Oct 31 12:35:07.370530 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:35:07.370496    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19431d95_2890_4c4c_b7b4_d7adb4fd51d5.slice/crio-0e1b12d4b64144c1c8a5e30bf68daf838783cf8d289e8a17244e0992cb257472 WatchSource:0}: Error finding container 0e1b12d4b64144c1c8a5e30bf68daf838783cf8d289e8a17244e0992cb257472: Status 404 returned error can't find the container with id 0e1b12d4b64144c1c8a5e30bf68daf838783cf8d289e8a17244e0992cb257472
Oct 31 12:35:08.122281 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:08.122252    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx" event={"ID":"19431d95-2890-4c4c-b7b4-d7adb4fd51d5","Type":"ContainerStarted","Data":"0e1b12d4b64144c1c8a5e30bf68daf838783cf8d289e8a17244e0992cb257472"}
Oct 31 12:35:09.128036 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:09.127990    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx" event={"ID":"19431d95-2890-4c4c-b7b4-d7adb4fd51d5","Type":"ContainerStarted","Data":"6dd14032424e7ac550b0bcc4d2238bf6006460ca64231cde4f2cabbfc37f0ae1"}
Oct 31 12:35:09.128502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:09.128125    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"
Oct 31 12:35:09.148908 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:09.148858    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx" podStartSLOduration=2.426808404 podStartE2EDuration="3.148844383s" totalImagesPullingTime="722.035979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:35:09.147286746 +0000 UTC m=+433.887712962" watchObservedRunningTime="2026-10-31 12:35:09.148844383 +0000 UTC m=+433.889270599"
Oct 31 12:35:13.184134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.184083    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"]
Oct 31 12:35:13.184640 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.184301    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck" podUID="da963526-ab82-4672-84dc-21c68b7de535" containerName="maas-api" containerID="cri-o://b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff" gracePeriod=30
Oct 31 12:35:13.427245 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.427223    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:35:13.542738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.542655    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-maas-api-tls\") pod \"da963526-ab82-4672-84dc-21c68b7de535\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") "
Oct 31 12:35:13.542738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.542701    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-metrics-certs\") pod \"da963526-ab82-4672-84dc-21c68b7de535\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") "
Oct 31 12:35:13.542932 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.542774    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da963526-ab82-4672-84dc-21c68b7de535-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da963526-ab82-4672-84dc-21c68b7de535-openshift-service-ca\") pod \"da963526-ab82-4672-84dc-21c68b7de535\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") "
Oct 31 12:35:13.542932 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.542828    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da963526-ab82-4672-84dc-21c68b7de535-kube-api-access-2qtjn\" (UniqueName: \"kubernetes.io/projected/da963526-ab82-4672-84dc-21c68b7de535-kube-api-access-2qtjn\") pod \"da963526-ab82-4672-84dc-21c68b7de535\" (UID: \"da963526-ab82-4672-84dc-21c68b7de535\") "
Oct 31 12:35:13.543209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.543176    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da963526-ab82-4672-84dc-21c68b7de535-openshift-service-ca" pod "da963526-ab82-4672-84dc-21c68b7de535" (UID: "da963526-ab82-4672-84dc-21c68b7de535"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:35:13.545001 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.544977    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-maas-api-tls" pod "da963526-ab82-4672-84dc-21c68b7de535" (UID: "da963526-ab82-4672-84dc-21c68b7de535"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:35:13.545079 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.544997    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-metrics-certs" pod "da963526-ab82-4672-84dc-21c68b7de535" (UID: "da963526-ab82-4672-84dc-21c68b7de535"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:35:13.545079 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.545069    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da963526-ab82-4672-84dc-21c68b7de535-kube-api-access-2qtjn" pod "da963526-ab82-4672-84dc-21c68b7de535" (UID: "da963526-ab82-4672-84dc-21c68b7de535"). InnerVolumeSpecName "kube-api-access-2qtjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:35:13.643579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.643541    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:35:13.643579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.643568    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da963526-ab82-4672-84dc-21c68b7de535-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:35:13.643579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.643579    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da963526-ab82-4672-84dc-21c68b7de535-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:35:13.643852 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:13.643590    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qtjn\" (UniqueName: \"kubernetes.io/projected/da963526-ab82-4672-84dc-21c68b7de535-kube-api-access-2qtjn\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:35:14.150100 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:14.150063    5663 generic.go:356] "Generic (PLEG): container finished" podID="da963526-ab82-4672-84dc-21c68b7de535" containerID="b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff" exitCode=0
Oct 31 12:35:14.150249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:14.150109    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"
Oct 31 12:35:14.150249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:14.150153    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck" event={"ID":"da963526-ab82-4672-84dc-21c68b7de535","Type":"ContainerDied","Data":"b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff"}
Oct 31 12:35:14.150249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:14.150208    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck" event={"ID":"da963526-ab82-4672-84dc-21c68b7de535","Type":"ContainerDied","Data":"252e252eff26c219aae16cceda03c0317003609f06bd791703c375a4f34ba62f"}
Oct 31 12:35:14.150249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:14.150231    5663 scope.go:122] "RemoveContainer" containerID="b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff"
Oct 31 12:35:14.159734 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:14.159715    5663 scope.go:122] "RemoveContainer" containerID="b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff"
Oct 31 12:35:14.159993 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:35:14.159974    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff\": container with ID starting with b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff not found: ID does not exist" containerID="b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff"
Oct 31 12:35:14.160069 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:14.160009    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff"} err="failed to get container status \"b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff\": rpc error: code = NotFound desc = could not find container \"b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff\": container with ID starting with b63286dfdc9116fbe62f1889bfb0073727f46340b93bd44a86926a7c84f3f2ff not found: ID does not exist"
Oct 31 12:35:14.175291 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:14.175267    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"]
Oct 31 12:35:14.178731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:14.178710    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-bg7ck"]
Oct 31 12:35:15.437379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:15.437333    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da963526-ab82-4672-84dc-21c68b7de535" path="/var/lib/kubelet/pods/da963526-ab82-4672-84dc-21c68b7de535/volumes"
Oct 31 12:35:22.474950 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:35:22.474913    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:35:40.135614 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:40.135532    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mt6xx"
Oct 31 12:35:40.183169 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:40.183138    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"]
Oct 31 12:35:40.183421 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:40.183386    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf" podUID="4fe387a8-f75e-4acb-973d-06f2bf25c8be" containerName="manager" containerID="cri-o://56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357" gracePeriod=30
Oct 31 12:35:40.432086 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:40.432056    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"
Oct 31 12:35:40.468716 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:40.468686    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe387a8-f75e-4acb-973d-06f2bf25c8be-kube-api-access-772z8\" (UniqueName: \"kubernetes.io/projected/4fe387a8-f75e-4acb-973d-06f2bf25c8be-kube-api-access-772z8\") pod \"4fe387a8-f75e-4acb-973d-06f2bf25c8be\" (UID: \"4fe387a8-f75e-4acb-973d-06f2bf25c8be\") "
Oct 31 12:35:40.468863 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:40.468723    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe387a8-f75e-4acb-973d-06f2bf25c8be-cert\" (UniqueName: \"kubernetes.io/secret/4fe387a8-f75e-4acb-973d-06f2bf25c8be-cert\") pod \"4fe387a8-f75e-4acb-973d-06f2bf25c8be\" (UID: \"4fe387a8-f75e-4acb-973d-06f2bf25c8be\") "
Oct 31 12:35:40.470841 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:40.470809    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe387a8-f75e-4acb-973d-06f2bf25c8be-kube-api-access-772z8" pod "4fe387a8-f75e-4acb-973d-06f2bf25c8be" (UID: "4fe387a8-f75e-4acb-973d-06f2bf25c8be"). InnerVolumeSpecName "kube-api-access-772z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:35:40.470949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:40.470882    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe387a8-f75e-4acb-973d-06f2bf25c8be-cert" pod "4fe387a8-f75e-4acb-973d-06f2bf25c8be" (UID: "4fe387a8-f75e-4acb-973d-06f2bf25c8be"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:35:40.570220 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:40.570185    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-772z8\" (UniqueName: \"kubernetes.io/projected/4fe387a8-f75e-4acb-973d-06f2bf25c8be-kube-api-access-772z8\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:35:40.570220 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:40.570219    5663 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fe387a8-f75e-4acb-973d-06f2bf25c8be-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:35:41.266303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:41.266267    5663 generic.go:356] "Generic (PLEG): container finished" podID="4fe387a8-f75e-4acb-973d-06f2bf25c8be" containerID="56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357" exitCode=0
Oct 31 12:35:41.266796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:41.266317    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"
Oct 31 12:35:41.266796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:41.266370    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf" event={"ID":"4fe387a8-f75e-4acb-973d-06f2bf25c8be","Type":"ContainerDied","Data":"56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357"}
Oct 31 12:35:41.266796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:41.266438    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf" event={"ID":"4fe387a8-f75e-4acb-973d-06f2bf25c8be","Type":"ContainerDied","Data":"9bfd00ba9cfa6bbfca6407680d5ce0f3670f01f2cef3ec46637893decc3a8ea8"}
Oct 31 12:35:41.266796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:41.266477    5663 scope.go:122] "RemoveContainer" containerID="56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357"
Oct 31 12:35:41.279634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:41.279609    5663 scope.go:122] "RemoveContainer" containerID="56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357"
Oct 31 12:35:41.279908 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:35:41.279890    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357\": container with ID starting with 56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357 not found: ID does not exist" containerID="56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357"
Oct 31 12:35:41.279973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:41.279916    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357"} err="failed to get container status \"56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357\": rpc error: code = NotFound desc = could not find container \"56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357\": container with ID starting with 56294078793f90d54c7815ac7031b9009d2d236339ebbf25f9b5bda235a6c357 not found: ID does not exist"
Oct 31 12:35:41.308516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:41.308492    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"]
Oct 31 12:35:41.326731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:41.326709    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5cqdf"]
Oct 31 12:35:41.436259 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:35:41.436227    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe387a8-f75e-4acb-973d-06f2bf25c8be" path="/var/lib/kubelet/pods/4fe387a8-f75e-4acb-973d-06f2bf25c8be/volumes"
Oct 31 12:35:52.475085 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:35:52.475050    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:36:22.475695 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:36:22.475653    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:36:29.224100 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.224067    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"]
Oct 31 12:36:29.224584 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.224429    5663 cpu_manager.go:414] "containerMap: removing container" podUID="da963526-ab82-4672-84dc-21c68b7de535" containerName="maas-api"
Oct 31 12:36:29.224584 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.224441    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da963526-ab82-4672-84dc-21c68b7de535" containerName="maas-api"
Oct 31 12:36:29.224584 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.224455    5663 cpu_manager.go:414] "containerMap: removing container" podUID="4fe387a8-f75e-4acb-973d-06f2bf25c8be" containerName="manager"
Oct 31 12:36:29.224584 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.224460    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe387a8-f75e-4acb-973d-06f2bf25c8be" containerName="manager"
Oct 31 12:36:29.224584 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.224506    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="da963526-ab82-4672-84dc-21c68b7de535" containerName="maas-api"
Oct 31 12:36:29.224584 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.224515    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe387a8-f75e-4acb-973d-06f2bf25c8be" containerName="manager"
Oct 31 12:36:29.227605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.227589    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.249497 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.249472    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"]
Oct 31 12:36:29.322106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.322079    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f7jrv\""
Oct 31 12:36:29.395791 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.395749    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.395791 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.395792    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.396008 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.395815    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.396008 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.395919    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.396008 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.395988    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.396008 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.396006    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.396145 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.396034    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.396145 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.396074    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.396145 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.396135    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57rbq\" (UniqueName: \"kubernetes.io/projected/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-kube-api-access-57rbq\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.497595 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.497511    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.497595 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.497557    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57rbq\" (UniqueName: \"kubernetes.io/projected/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-kube-api-access-57rbq\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.497825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.497602    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.497825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.497619    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.497825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.497640    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.497825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.497660    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.497825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.497682    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.498067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.497836    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.498067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.497891    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.498173 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.498101    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.498173 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.498126    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.498293 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.498222    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.498349 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.498327    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.498788 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.498764    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.499912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.499894    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.500199 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.500181    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.510539 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.510515    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.510642 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.510593    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57rbq\" (UniqueName: \"kubernetes.io/projected/6cacf32b-0e2d-48cc-a0be-c99e26159bcc-kube-api-access-57rbq\") pod \"maas-default-gateway-openshift-default-645cf464d7-lsjpx\" (UID: \"6cacf32b-0e2d-48cc-a0be-c99e26159bcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.540460 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.540425    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:29.821138 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:29.821104    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"]
Oct 31 12:36:29.823880 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:36:29.823852    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cacf32b_0e2d_48cc_a0be_c99e26159bcc.slice/crio-46d46730ae95751319ddc56acb021e690db998879c00a910d3bb66fc19959d5a WatchSource:0}: Error finding container 46d46730ae95751319ddc56acb021e690db998879c00a910d3bb66fc19959d5a: Status 404 returned error can't find the container with id 46d46730ae95751319ddc56acb021e690db998879c00a910d3bb66fc19959d5a
Oct 31 12:36:30.465628 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:30.465582    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx" event={"ID":"6cacf32b-0e2d-48cc-a0be-c99e26159bcc","Type":"ContainerStarted","Data":"46d46730ae95751319ddc56acb021e690db998879c00a910d3bb66fc19959d5a"}
Oct 31 12:36:33.945151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:33.945119    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:36:33.945428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:33.945190    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:36:33.945428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:33.945221    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:36:34.486094 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:34.486058    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx" event={"ID":"6cacf32b-0e2d-48cc-a0be-c99e26159bcc","Type":"ContainerStarted","Data":"2b3930a5674ef477f5b7c628c35a24a84222475dbda66d65a4460162371fec47"}
Oct 31 12:36:34.514479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:34.514425    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx" podStartSLOduration=1.395966353 podStartE2EDuration="5.514411487s" totalImagesPullingTime="4.118445134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:34.512510366 +0000 UTC m=+519.252936582" watchObservedRunningTime="2026-10-31 12:36:34.514411487 +0000 UTC m=+519.254837703"
Oct 31 12:36:34.540699 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:34.540649    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:35.542871 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:35.542839    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-lsjpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 12:36:35.543326 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:35.542897    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx" podUID="6cacf32b-0e2d-48cc-a0be-c99e26159bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 12:36:36.541027 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:36.540987    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-lsjpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 12:36:36.541203 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:36.541042    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx" podUID="6cacf32b-0e2d-48cc-a0be-c99e26159bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 12:36:37.541335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:37.541296    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-lsjpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 12:36:37.541765 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:37.541381    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx" podUID="6cacf32b-0e2d-48cc-a0be-c99e26159bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 12:36:38.541412 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:38.541378    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-lsjpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 12:36:38.541792 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:38.541433    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx" podUID="6cacf32b-0e2d-48cc-a0be-c99e26159bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 12:36:39.541669 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:39.541631    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-lsjpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 12:36:39.542078 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:39.541691    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx" podUID="6cacf32b-0e2d-48cc-a0be-c99e26159bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 12:36:40.541342 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:40.541307    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-lsjpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 12:36:40.541549 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:40.541387    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx" podUID="6cacf32b-0e2d-48cc-a0be-c99e26159bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 12:36:42.202375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:42.202307    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:42.204976 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:42.203519    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:42.205252 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:42.205227    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx"
Oct 31 12:36:44.230032 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.229998    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"]
Oct 31 12:36:44.233544 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.233515    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.246517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.246493    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"]
Oct 31 12:36:44.355964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.355932    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.356146 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.355971    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.356146 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.355996    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.356146 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.356101    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.356146 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.356118    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.356146 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.356139    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353c883f-02b3-49e1-83bf-e874321bfcca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.356401 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.356177    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.356401 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.356267    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79g4\" (UniqueName: \"kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-kube-api-access-c79g4\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.356401 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.356295    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353c883f-02b3-49e1-83bf-e874321bfcca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457140 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457107    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457140 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457141    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457162    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457223    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457276    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457304    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353c883f-02b3-49e1-83bf-e874321bfcca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457582 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457383    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457582 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457445    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79g4\" (UniqueName: \"kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-kube-api-access-c79g4\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457582 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457474    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353c883f-02b3-49e1-83bf-e874321bfcca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457582 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457542    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457707    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457855 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457816    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.457921 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457901    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.458019 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.457998    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353c883f-02b3-49e1-83bf-e874321bfcca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.459437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.459412    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.459873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.459855    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353c883f-02b3-49e1-83bf-e874321bfcca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.469761 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.469731    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79g4\" (UniqueName: \"kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-kube-api-access-c79g4\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.469858 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.469799    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-ms7fp\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.547644 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.547560    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:44.826657 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.826622    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"]
Oct 31 12:36:44.830511 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:36:44.830459    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353c883f_02b3_49e1_83bf_e874321bfcca.slice/crio-77e2f0627d8a1f851887bb7266355614a91de9953160c3da616079b3a4a28de1 WatchSource:0}: Error finding container 77e2f0627d8a1f851887bb7266355614a91de9953160c3da616079b3a4a28de1: Status 404 returned error can't find the container with id 77e2f0627d8a1f851887bb7266355614a91de9953160c3da616079b3a4a28de1
Oct 31 12:36:44.833450 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.833418    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:36:44.833516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.833485    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:36:44.833555 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:44.833528    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:36:45.530597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:45.530566    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" event={"ID":"353c883f-02b3-49e1-83bf-e874321bfcca","Type":"ContainerStarted","Data":"b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e"}
Oct 31 12:36:45.531029 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:45.530614    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" event={"ID":"353c883f-02b3-49e1-83bf-e874321bfcca","Type":"ContainerStarted","Data":"77e2f0627d8a1f851887bb7266355614a91de9953160c3da616079b3a4a28de1"}
Oct 31 12:36:45.548398 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:45.548373    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:45.549770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:45.549747    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-ms7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 12:36:45.549883 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:45.549799    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 12:36:45.555707 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:45.555666    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" podStartSLOduration=1.555654954 podStartE2EDuration="1.555654954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:36:45.554575784 +0000 UTC m=+530.295002007" watchObservedRunningTime="2026-10-31 12:36:45.555654954 +0000 UTC m=+530.296081171"
Oct 31 12:36:46.548646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:46.548617    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-ms7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 12:36:46.549091 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:46.548663    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 12:36:47.547921 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:47.547883    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-ms7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 12:36:47.548100 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:47.547954    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 12:36:48.548443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:48.548412    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-ms7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 12:36:48.548792 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:48.548458    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 12:36:49.548903 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:49.548868    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-ms7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 12:36:49.549306 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:49.548917    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 12:36:50.548159 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:50.548125    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-ms7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 12:36:50.548390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:50.548186    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 12:36:51.548250 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:51.548215    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-ms7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 31 12:36:51.548622 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:51.548265    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 31 12:36:52.475283 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:36:52.475243    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:36:53.045708 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:53.045669    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:53.046640 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:53.046620    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:36:53.047483 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:36:53.047461    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:37:22.481050 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:37:22.480992    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:37:27.713439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.713404    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"]
Oct 31 12:37:27.720647 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.720630    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:27.730745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.730717    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-fb0343-serving-cert\""
Oct 31 12:37:27.730873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.730758    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 12:37:27.732034 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.732010    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"]
Oct 31 12:37:27.732894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.732876    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6jkzx\""
Oct 31 12:37:27.752741 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.752706    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-metrics-certs\") pod \"maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:27.752838 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.752761    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szzfh\" (UniqueName: \"kubernetes.io/projected/096ba82b-2c3c-439c-8010-b876ea2319a8-kube-api-access-szzfh\") pod \"maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:27.752886 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.752849    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096ba82b-2c3c-439c-8010-b876ea2319a8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:27.752921 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.752894    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:27.854037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.854004    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-metrics-certs\") pod \"maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:27.854037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.854042    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szzfh\" (UniqueName: \"kubernetes.io/projected/096ba82b-2c3c-439c-8010-b876ea2319a8-kube-api-access-szzfh\") pod \"maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:27.854282 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.854068    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096ba82b-2c3c-439c-8010-b876ea2319a8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:27.854282 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.854096    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:27.854739 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.854705    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096ba82b-2c3c-439c-8010-b876ea2319a8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:27.856576 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.856554    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-metrics-certs\") pod \"maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:27.856679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.856612    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:27.919504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:27.919470    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szzfh\" (UniqueName: \"kubernetes.io/projected/096ba82b-2c3c-439c-8010-b876ea2319a8-kube-api-access-szzfh\") pod \"maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:28.032842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:28.032746    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:28.310489 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:28.310458    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"]
Oct 31 12:37:28.313471 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:37:28.313445    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096ba82b_2c3c_439c_8010_b876ea2319a8.slice/crio-ba219a1000fc6de7a5ae74be7e024e446c4fe37b39ba9dd3cc5017b905b6397f WatchSource:0}: Error finding container ba219a1000fc6de7a5ae74be7e024e446c4fe37b39ba9dd3cc5017b905b6397f: Status 404 returned error can't find the container with id ba219a1000fc6de7a5ae74be7e024e446c4fe37b39ba9dd3cc5017b905b6397f
Oct 31 12:37:28.711071 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:28.710990    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9" event={"ID":"096ba82b-2c3c-439c-8010-b876ea2319a8","Type":"ContainerStarted","Data":"ba219a1000fc6de7a5ae74be7e024e446c4fe37b39ba9dd3cc5017b905b6397f"}
Oct 31 12:37:29.716933 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:29.716898    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9" event={"ID":"096ba82b-2c3c-439c-8010-b876ea2319a8","Type":"ContainerStarted","Data":"ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db"}
Oct 31 12:37:29.717472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:29.717004    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:29.737593 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:29.737550    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9" podStartSLOduration=2.2543125760000002 podStartE2EDuration="2.737535794s" totalImagesPullingTime="483.223218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:29.735683942 +0000 UTC m=+574.476110158" watchObservedRunningTime="2026-10-31 12:37:29.737535794 +0000 UTC m=+574.477962009"
Oct 31 12:37:30.080756 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.080721    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"]
Oct 31 12:37:30.083686 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.083671    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"
Oct 31 12:37:30.089242 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.089219    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-b25d3b-dockercfg-g2knq\""
Oct 31 12:37:30.091537 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.091513    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-b25d3b\""
Oct 31 12:37:30.096664 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.096643    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"]
Oct 31 12:37:30.181417 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.181378    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnms2\" (UniqueName: \"kubernetes.io/projected/d279e306-b346-462a-a120-9fb77b6dcb0b-kube-api-access-vnms2\") pod \"payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x\" (UID: \"d279e306-b346-462a-a120-9fb77b6dcb0b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"
Oct 31 12:37:30.181618 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.181489    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d279e306-b346-462a-a120-9fb77b6dcb0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x\" (UID: \"d279e306-b346-462a-a120-9fb77b6dcb0b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"
Oct 31 12:37:30.281969 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.281933    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d279e306-b346-462a-a120-9fb77b6dcb0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x\" (UID: \"d279e306-b346-462a-a120-9fb77b6dcb0b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"
Oct 31 12:37:30.282133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.282016    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnms2\" (UniqueName: \"kubernetes.io/projected/d279e306-b346-462a-a120-9fb77b6dcb0b-kube-api-access-vnms2\") pod \"payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x\" (UID: \"d279e306-b346-462a-a120-9fb77b6dcb0b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"
Oct 31 12:37:30.282672 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.282652    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d279e306-b346-462a-a120-9fb77b6dcb0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x\" (UID: \"d279e306-b346-462a-a120-9fb77b6dcb0b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"
Oct 31 12:37:30.294455 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.294422    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnms2\" (UniqueName: \"kubernetes.io/projected/d279e306-b346-462a-a120-9fb77b6dcb0b-kube-api-access-vnms2\") pod \"payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x\" (UID: \"d279e306-b346-462a-a120-9fb77b6dcb0b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"
Oct 31 12:37:30.396997 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.396913    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"
Oct 31 12:37:30.667616 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.667562    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"]
Oct 31 12:37:30.670113 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:37:30.670086    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd279e306_b346_462a_a120_9fb77b6dcb0b.slice/crio-b0261b082d4c2e69dba8cf408a392444f8d17404f4bd4ce0872e2ad7d128dae8 WatchSource:0}: Error finding container b0261b082d4c2e69dba8cf408a392444f8d17404f4bd4ce0872e2ad7d128dae8: Status 404 returned error can't find the container with id b0261b082d4c2e69dba8cf408a392444f8d17404f4bd4ce0872e2ad7d128dae8
Oct 31 12:37:30.721797 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.721766    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x" event={"ID":"d279e306-b346-462a-a120-9fb77b6dcb0b","Type":"ContainerStarted","Data":"b0261b082d4c2e69dba8cf408a392444f8d17404f4bd4ce0872e2ad7d128dae8"}
Oct 31 12:37:30.751490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.751459    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"]
Oct 31 12:37:30.756048 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.756019    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"
Oct 31 12:37:30.781508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.781483    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e6c79d-dockercfg-t25fz\""
Oct 31 12:37:30.782033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.782008    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e6c79d\""
Oct 31 12:37:30.787380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.787333    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkdq6\" (UniqueName: \"kubernetes.io/projected/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-kube-api-access-tkdq6\") pod \"payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8\" (UID: \"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"
Oct 31 12:37:30.787506 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.787402    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8\" (UID: \"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"
Oct 31 12:37:30.800238 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.800204    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"]
Oct 31 12:37:30.888291 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.888253    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8\" (UID: \"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"
Oct 31 12:37:30.888494 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.888372    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkdq6\" (UniqueName: \"kubernetes.io/projected/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-kube-api-access-tkdq6\") pod \"payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8\" (UID: \"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"
Oct 31 12:37:30.889237 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.889211    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8\" (UID: \"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"
Oct 31 12:37:30.918282 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:30.918196    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkdq6\" (UniqueName: \"kubernetes.io/projected/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-kube-api-access-tkdq6\") pod \"payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8\" (UID: \"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"
Oct 31 12:37:31.068752 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.068715    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"
Oct 31 12:37:31.268541 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.268423    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"]
Oct 31 12:37:31.272677 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.272619    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"
Oct 31 12:37:31.284170 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.284140    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-85ce08\""
Oct 31 12:37:31.290340 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.290310    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-85ce08-dockercfg-m7ndj\""
Oct 31 12:37:31.290916 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.290893    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"]
Oct 31 12:37:31.352092 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.352058    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"]
Oct 31 12:37:31.355377 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:37:31.355334    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06c9a3d_a297_4211_aa4d_73ab00ec9aa0.slice/crio-ac53ecb01a604ff047bbdf4bcbd93e12caa80a312121eb239279a88b8af38068 WatchSource:0}: Error finding container ac53ecb01a604ff047bbdf4bcbd93e12caa80a312121eb239279a88b8af38068: Status 404 returned error can't find the container with id ac53ecb01a604ff047bbdf4bcbd93e12caa80a312121eb239279a88b8af38068
Oct 31 12:37:31.393112 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.393079    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a2a833-a8a0-499d-8784-310253f03010-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx\" (UID: \"97a2a833-a8a0-499d-8784-310253f03010\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"
Oct 31 12:37:31.393263 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.393140    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfrw6\" (UniqueName: \"kubernetes.io/projected/97a2a833-a8a0-499d-8784-310253f03010-kube-api-access-bfrw6\") pod \"payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx\" (UID: \"97a2a833-a8a0-499d-8784-310253f03010\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"
Oct 31 12:37:31.494397 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.494339    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a2a833-a8a0-499d-8784-310253f03010-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx\" (UID: \"97a2a833-a8a0-499d-8784-310253f03010\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"
Oct 31 12:37:31.494597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.494439    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfrw6\" (UniqueName: \"kubernetes.io/projected/97a2a833-a8a0-499d-8784-310253f03010-kube-api-access-bfrw6\") pod \"payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx\" (UID: \"97a2a833-a8a0-499d-8784-310253f03010\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"
Oct 31 12:37:31.495252 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.495229    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a2a833-a8a0-499d-8784-310253f03010-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx\" (UID: \"97a2a833-a8a0-499d-8784-310253f03010\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"
Oct 31 12:37:31.508989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.508958    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfrw6\" (UniqueName: \"kubernetes.io/projected/97a2a833-a8a0-499d-8784-310253f03010-kube-api-access-bfrw6\") pod \"payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx\" (UID: \"97a2a833-a8a0-499d-8784-310253f03010\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"
Oct 31 12:37:31.602868 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.602824    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"
Oct 31 12:37:31.728927 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.728876    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8" event={"ID":"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0","Type":"ContainerStarted","Data":"ac53ecb01a604ff047bbdf4bcbd93e12caa80a312121eb239279a88b8af38068"}
Oct 31 12:37:31.911348 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:31.911259    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"]
Oct 31 12:37:31.917876 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:37:31.917838    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a2a833_a8a0_499d_8784_310253f03010.slice/crio-d4777435abd8a32d0df77e640e774ed437669052e86b1a3e81076ff0d806d0c1 WatchSource:0}: Error finding container d4777435abd8a32d0df77e640e774ed437669052e86b1a3e81076ff0d806d0c1: Status 404 returned error can't find the container with id d4777435abd8a32d0df77e640e774ed437669052e86b1a3e81076ff0d806d0c1
Oct 31 12:37:32.249591 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.249490    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"]
Oct 31 12:37:32.260538 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.260500    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.262086 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.262062    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"]
Oct 31 12:37:32.265856 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.265834    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-67d410-serving-cert\""
Oct 31 12:37:32.302168 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.302131    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dkkm\" (UniqueName: \"kubernetes.io/projected/96e5a20b-4141-4cbe-89dd-888c80811889-kube-api-access-5dkkm\") pod \"maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.302345 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.302198    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-metrics-certs\") pod \"maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.302345 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.302266    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e5a20b-4141-4cbe-89dd-888c80811889-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.302345 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.302307    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-maas-api-tls\") pod \"maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.406414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.406292    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-maas-api-tls\") pod \"maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.408736 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.407605    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkkm\" (UniqueName: \"kubernetes.io/projected/96e5a20b-4141-4cbe-89dd-888c80811889-kube-api-access-5dkkm\") pod \"maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.408736 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.407665    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-metrics-certs\") pod \"maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.408736 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.407727    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e5a20b-4141-4cbe-89dd-888c80811889-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.409211 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.409162    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e5a20b-4141-4cbe-89dd-888c80811889-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.410654 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.410610    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-maas-api-tls\") pod \"maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.412153 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.412118    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-metrics-certs\") pod \"maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.427027 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.427002    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkkm\" (UniqueName: \"kubernetes.io/projected/96e5a20b-4141-4cbe-89dd-888c80811889-kube-api-access-5dkkm\") pod \"maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.578263 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.578229    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:32.735993 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.735960    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx" event={"ID":"97a2a833-a8a0-499d-8784-310253f03010","Type":"ContainerStarted","Data":"d4777435abd8a32d0df77e640e774ed437669052e86b1a3e81076ff0d806d0c1"}
Oct 31 12:37:32.881124 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:32.881099    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"]
Oct 31 12:37:33.744624 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:33.744593    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x" event={"ID":"d279e306-b346-462a-a120-9fb77b6dcb0b","Type":"ContainerStarted","Data":"68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3"}
Oct 31 12:37:33.745043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:33.744705    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"
Oct 31 12:37:33.746964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:33.746935    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb" event={"ID":"96e5a20b-4141-4cbe-89dd-888c80811889","Type":"ContainerStarted","Data":"c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42"}
Oct 31 12:37:33.747105 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:33.746983    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb" event={"ID":"96e5a20b-4141-4cbe-89dd-888c80811889","Type":"ContainerStarted","Data":"52f91cffce272050755638ed7af01dc6cd85f53c444bd01db8b5ca7c5711985c"}
Oct 31 12:37:33.747105 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:33.747024    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:33.771146 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:33.770156    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x" podStartSLOduration=0.842776961 podStartE2EDuration="3.77013974s" totalImagesPullingTime="2.927362779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:33.764474749 +0000 UTC m=+578.504900965" watchObservedRunningTime="2026-10-31 12:37:33.77013974 +0000 UTC m=+578.510565960"
Oct 31 12:37:33.785213 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:33.785165    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb" podStartSLOduration=1.303126741 podStartE2EDuration="1.785148582s" totalImagesPullingTime="482.021841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:33.784969604 +0000 UTC m=+578.525395822" watchObservedRunningTime="2026-10-31 12:37:33.785148582 +0000 UTC m=+578.525574789"
Oct 31 12:37:34.752714 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:34.752677    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8" event={"ID":"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0","Type":"ContainerStarted","Data":"96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e"}
Oct 31 12:37:34.753235 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:34.752766    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"
Oct 31 12:37:34.756937 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:34.756555    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx" event={"ID":"97a2a833-a8a0-499d-8784-310253f03010","Type":"ContainerStarted","Data":"77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011"}
Oct 31 12:37:34.757401 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:34.757377    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"
Oct 31 12:37:34.772406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:34.772340    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8" podStartSLOduration=2.138344958 podStartE2EDuration="4.772324642s" totalImagesPullingTime="2.633979684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:34.771079969 +0000 UTC m=+579.511506185" watchObservedRunningTime="2026-10-31 12:37:34.772324642 +0000 UTC m=+579.512750857"
Oct 31 12:37:34.788322 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:34.788268    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx" podStartSLOduration=2.001697541 podStartE2EDuration="3.788257816s" totalImagesPullingTime="1.786560275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:34.787872547 +0000 UTC m=+579.528298785" watchObservedRunningTime="2026-10-31 12:37:34.788257816 +0000 UTC m=+579.528684033"
Oct 31 12:37:35.651675 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.651640    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"]
Oct 31 12:37:35.654887 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.654864    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.663251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.663227    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-b218d3-serving-cert\""
Oct 31 12:37:35.666289 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.666210    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"]
Oct 31 12:37:35.727488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.727464    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:37:35.756365 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.756321    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb2nm\" (UniqueName: \"kubernetes.io/projected/fa58022c-031e-45fb-9f00-d188fca88d48-kube-api-access-wb2nm\") pod \"maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.756723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.756449    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa58022c-031e-45fb-9f00-d188fca88d48-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.756723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.756492    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-metrics-certs\") pod \"maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.756723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.756523    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.857307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.857266    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb2nm\" (UniqueName: \"kubernetes.io/projected/fa58022c-031e-45fb-9f00-d188fca88d48-kube-api-access-wb2nm\") pod \"maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.857530 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.857384    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa58022c-031e-45fb-9f00-d188fca88d48-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.857530 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.857442    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-metrics-certs\") pod \"maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.857530 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.857526    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.858155 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.858128    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa58022c-031e-45fb-9f00-d188fca88d48-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.860221 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.860193    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.860370 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.860330    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-metrics-certs\") pod \"maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.874089 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.873761    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb2nm\" (UniqueName: \"kubernetes.io/projected/fa58022c-031e-45fb-9f00-d188fca88d48-kube-api-access-wb2nm\") pod \"maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:35.971671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:35.971579    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:36.255871 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:36.250725    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"]
Oct 31 12:37:36.258847 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:37:36.258809    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa58022c_031e_45fb_9f00_d188fca88d48.slice/crio-0c5e40c3dd6cb708cc2b24d006bd06d6bda830ac30ef8145ee25f02c2937979a WatchSource:0}: Error finding container 0c5e40c3dd6cb708cc2b24d006bd06d6bda830ac30ef8145ee25f02c2937979a: Status 404 returned error can't find the container with id 0c5e40c3dd6cb708cc2b24d006bd06d6bda830ac30ef8145ee25f02c2937979a
Oct 31 12:37:36.768701 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:36.768659    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz" event={"ID":"fa58022c-031e-45fb-9f00-d188fca88d48","Type":"ContainerStarted","Data":"37743c0919a42cdddaad2fb2c29b75836a06c217630baaa001cef79ab77b1450"}
Oct 31 12:37:36.769191 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:36.768725    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz" event={"ID":"fa58022c-031e-45fb-9f00-d188fca88d48","Type":"ContainerStarted","Data":"0c5e40c3dd6cb708cc2b24d006bd06d6bda830ac30ef8145ee25f02c2937979a"}
Oct 31 12:37:36.769191 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:36.768985    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:36.789065 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:36.789018    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz" podStartSLOduration=1.3907401 podStartE2EDuration="1.789002717s" totalImagesPullingTime="398.262617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:36.787579777 +0000 UTC m=+581.528005992" watchObservedRunningTime="2026-10-31 12:37:36.789002717 +0000 UTC m=+581.529428932"
Oct 31 12:37:39.763025 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:39.762995    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 12:37:42.778290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:42.778262    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:37:44.614196 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.614159    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"]
Oct 31 12:37:44.617901 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.617880    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.623670 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.623646    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"openshift-service-ca.crt\""
Oct 31 12:37:44.627448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.626667    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"kube-root-ca.crt\""
Oct 31 12:37:44.627843 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.627819    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"simulator-w5-b25d3b-kserve-self-signed-certs\""
Oct 31 12:37:44.629585 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.629564    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"]
Oct 31 12:37:44.629689 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.629677    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"default-dockercfg-rt29c\""
Oct 31 12:37:44.754375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.754321    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-home\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.754547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.754479    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tls-certs\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.754547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.754508    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-dshm\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.754547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.754527    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8wqn\" (UniqueName: \"kubernetes.io/projected/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-kube-api-access-b8wqn\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.754649 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.754576    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-model-cache\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.754694 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.754673    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tmp-dir\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.760018 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.759995    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"
Oct 31 12:37:44.855283 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.855252    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-home\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.855462 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.855452    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tls-certs\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.855518 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.855479    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-dshm\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.855518 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.855496    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wqn\" (UniqueName: \"kubernetes.io/projected/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-kube-api-access-b8wqn\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.855605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.855540    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-model-cache\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.855605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.855588    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tmp-dir\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.855734 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.855708    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-home\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.855916 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.855885    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tmp-dir\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.856017 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.855964    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-model-cache\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.857745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.857719    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-dshm\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.858042 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.858020    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tls-certs\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.868168 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.868112    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8wqn\" (UniqueName: \"kubernetes.io/projected/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-kube-api-access-b8wqn\") pod \"simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") " pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:44.949523 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:44.949488    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:45.229802 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:45.229743    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"]
Oct 31 12:37:45.232186 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:37:45.232158    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6546f0e_5df8_44d9_9ed1_d7a115a6441a.slice/crio-70e3dba285f80c9a0b7f010befd1f7492646d6eed0f64d20d14d43c196b0b23b WatchSource:0}: Error finding container 70e3dba285f80c9a0b7f010befd1f7492646d6eed0f64d20d14d43c196b0b23b: Status 404 returned error can't find the container with id 70e3dba285f80c9a0b7f010befd1f7492646d6eed0f64d20d14d43c196b0b23b
Oct 31 12:37:45.764486 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:45.764455    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"
Oct 31 12:37:45.811101 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:45.811057    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" event={"ID":"a6546f0e-5df8-44d9-9ed1-d7a115a6441a","Type":"ContainerStarted","Data":"70e3dba285f80c9a0b7f010befd1f7492646d6eed0f64d20d14d43c196b0b23b"}
Oct 31 12:37:46.772433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:46.772399    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"
Oct 31 12:37:46.818511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:46.818473    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" event={"ID":"a6546f0e-5df8-44d9-9ed1-d7a115a6441a","Type":"ContainerStarted","Data":"e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74"}
Oct 31 12:37:46.850511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:46.850461    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podStartSLOduration=1.474780911 podStartE2EDuration="2.850444236s" totalImagesPullingTime="1.375663325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:46.849158609 +0000 UTC m=+591.589584860" watchObservedRunningTime="2026-10-31 12:37:46.850444236 +0000 UTC m=+591.590870452"
Oct 31 12:37:46.949841 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:46.949761    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:52.456926 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.456896    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"]
Oct 31 12:37:52.463476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.463453    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.480677 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:37:52.480646    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:37:52.481158 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.481139    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"]
Oct 31 12:37:52.482319 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.482303    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-67d410\"/\"default-dockercfg-jmw9g\""
Oct 31 12:37:52.485389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.482549    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-67d410\"/\"simulator-w6-67d410-kserve-self-signed-certs\""
Oct 31 12:37:52.540068 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.540024    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-model-cache\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.540251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.540194    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tmp-dir\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.540251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.540237    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tls-certs\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.540456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.540313    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwswq\" (UniqueName: \"kubernetes.io/projected/5ab6a659-425e-493a-b02d-9a1a23c06aa1-kube-api-access-zwswq\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.540456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.540395    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-dshm\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.540456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.540422    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-home\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.555057 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.555024    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-67d410\"/\"kube-root-ca.crt\""
Oct 31 12:37:52.555288 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.555268    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-67d410\"/\"openshift-service-ca.crt\""
Oct 31 12:37:52.642389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.641129    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-dshm\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.642389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.641184    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-home\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.642389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.641265    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-model-cache\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.642389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.641347    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tmp-dir\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.642389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.641395    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tls-certs\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.642389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.641422    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwswq\" (UniqueName: \"kubernetes.io/projected/5ab6a659-425e-493a-b02d-9a1a23c06aa1-kube-api-access-zwswq\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.643379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.643340    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-model-cache\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.643950 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.643905    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-home\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.645715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.645694    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tmp-dir\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.661566 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.656098    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tls-certs\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.661566 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.660034    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwswq\" (UniqueName: \"kubernetes.io/projected/5ab6a659-425e-493a-b02d-9a1a23c06aa1-kube-api-access-zwswq\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.665774 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.665729    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-dshm\") pod \"simulator-w6-67d410-kserve-8588f46bcf-8b2ch\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") " pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:52.777309 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:52.777019    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:53.061128 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:53.061098    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"]
Oct 31 12:37:53.064483 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:37:53.064456    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ab6a659_425e_493a_b02d_9a1a23c06aa1.slice/crio-b6cde3d6f8de4c78b88e79cf887b591c4ead319c10208a44a79c6f4d0f1a4ae9 WatchSource:0}: Error finding container b6cde3d6f8de4c78b88e79cf887b591c4ead319c10208a44a79c6f4d0f1a4ae9: Status 404 returned error can't find the container with id b6cde3d6f8de4c78b88e79cf887b591c4ead319c10208a44a79c6f4d0f1a4ae9
Oct 31 12:37:53.860934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:53.860891    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" event={"ID":"5ab6a659-425e-493a-b02d-9a1a23c06aa1","Type":"ContainerStarted","Data":"380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761"}
Oct 31 12:37:53.861412 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:53.860945    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" event={"ID":"5ab6a659-425e-493a-b02d-9a1a23c06aa1","Type":"ContainerStarted","Data":"b6cde3d6f8de4c78b88e79cf887b591c4ead319c10208a44a79c6f4d0f1a4ae9"}
Oct 31 12:37:53.881046 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:53.881001    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podStartSLOduration=1.88098956 podStartE2EDuration="1.88098956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:53.879596281 +0000 UTC m=+598.620022497" watchObservedRunningTime="2026-10-31 12:37:53.88098956 +0000 UTC m=+598.621415775"
Oct 31 12:37:54.778514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:54.778482    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:37:54.950150 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:54.950111    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:54.963106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:54.963079    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:55.881695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:55.881665    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:37:57.063180 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.063138    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"]
Oct 31 12:37:57.069999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.069970    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.076628 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.076603    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"]
Oct 31 12:37:57.077930 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.077564    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-85ce08\"/\"kube-root-ca.crt\""
Oct 31 12:37:57.077930 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.077716    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-85ce08\"/\"simulator-w1-85ce08-kserve-self-signed-certs\""
Oct 31 12:37:57.078055 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.077940    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-85ce08\"/\"default-dockercfg-t7qtc\""
Oct 31 12:37:57.078055 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.078045    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-85ce08\"/\"openshift-service-ca.crt\""
Oct 31 12:37:57.083576 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.083550    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tls-certs\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.083708 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.083594    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-home\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.083708 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.083688    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-dshm\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.083842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.083816    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tmp-dir\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.083896 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.083860    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzm66\" (UniqueName: \"kubernetes.io/projected/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-kube-api-access-pzm66\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.083957 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.083940    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-model-cache\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.184868 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.184832    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tls-certs\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.185066 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.184878    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-home\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.185066 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.184909    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-dshm\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.185066 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.185053    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tmp-dir\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.185234 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.185087    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzm66\" (UniqueName: \"kubernetes.io/projected/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-kube-api-access-pzm66\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.185234 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.185154    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-model-cache\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.185429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.185396    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-home\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.185506 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.185453    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-model-cache\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.185798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.185771    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tmp-dir\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.187174 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.187155    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-dshm\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.187692 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.187670    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tls-certs\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.205276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.205250    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzm66\" (UniqueName: \"kubernetes.io/projected/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-kube-api-access-pzm66\") pod \"simulator-w1-85ce08-kserve-5766b7b6df-pzr4z\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") " pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.384722 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.384643    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:37:57.687418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.687320    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"]
Oct 31 12:37:57.690661 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:37:57.690630    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71a99b5e_57ec_41c2_9fdb_b380a0e4a152.slice/crio-0aabc58fa1533fa1613eac46810d8283f4d47f25c171602f53c3616ceb0b577f WatchSource:0}: Error finding container 0aabc58fa1533fa1613eac46810d8283f4d47f25c171602f53c3616ceb0b577f: Status 404 returned error can't find the container with id 0aabc58fa1533fa1613eac46810d8283f4d47f25c171602f53c3616ceb0b577f
Oct 31 12:37:57.881241 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.881202    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" event={"ID":"71a99b5e-57ec-41c2-9fdb-b380a0e4a152","Type":"ContainerStarted","Data":"b2e1410854f86d8ff1f0d049e59c1342f613356f64af72ea2cb7a07d19f81d72"}
Oct 31 12:37:57.881425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.881257    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" event={"ID":"71a99b5e-57ec-41c2-9fdb-b380a0e4a152","Type":"ContainerStarted","Data":"0aabc58fa1533fa1613eac46810d8283f4d47f25c171602f53c3616ceb0b577f"}
Oct 31 12:37:57.900648 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:57.900598    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podStartSLOduration=0.900585201 podStartE2EDuration="900.585201ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:37:57.89964324 +0000 UTC m=+602.640069470" watchObservedRunningTime="2026-10-31 12:37:57.900585201 +0000 UTC m=+602.641011453"
Oct 31 12:37:58.384766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:37:58.384731    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:38:02.777739 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:02.777703    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:38:02.790854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:02.790825    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:38:02.916523 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:02.916495    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:38:03.742281 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.742247    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"]
Oct 31 12:38:03.749267 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.749241    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.756672 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.756643    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b218d3\"/\"default-dockercfg-tqnrh\""
Oct 31 12:38:03.756815 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.756707    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b218d3\"/\"kube-root-ca.crt\""
Oct 31 12:38:03.756815 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.756737    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b218d3\"/\"openshift-service-ca.crt\""
Oct 31 12:38:03.756815 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.756802    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b218d3\"/\"simulator-w3-b218d3-kserve-self-signed-certs\""
Oct 31 12:38:03.758168 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.757913    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"]
Oct 31 12:38:03.849240 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.849205    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-dshm\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.849762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.849372    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jdb5\" (UniqueName: \"kubernetes.io/projected/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-kube-api-access-8jdb5\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.849762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.849433    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-home\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.849762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.849494    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tls-certs\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.849762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.849523    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tmp-dir\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.849762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.849543    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-model-cache\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.950502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.950456    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-home\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.950847 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.950826    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tls-certs\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.950972 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.950866    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tmp-dir\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.950972 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.950893    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-model-cache\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.950972 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.950934    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-home\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.950972 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.950963    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-dshm\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.951175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.951070    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jdb5\" (UniqueName: \"kubernetes.io/projected/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-kube-api-access-8jdb5\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.951639 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.951591    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-model-cache\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.952069 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.951964    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tmp-dir\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.955529 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.955504    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-dshm\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.955879 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.955852    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tls-certs\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:03.965133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:03.965108    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jdb5\" (UniqueName: \"kubernetes.io/projected/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-kube-api-access-8jdb5\") pod \"simulator-w3-b218d3-kserve-658d9b597d-lt2bf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") " pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:04.064876 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:04.064844    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:04.342103 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:04.342017    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"]
Oct 31 12:38:04.345722 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:38:04.345699    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f8e92eb_f0fa_4595_b18d_c9e78f360eaf.slice/crio-8093ef6feaf726a16768130f9f4cbbe6ff20cda96acf0d1b7145593b0f28ebf8 WatchSource:0}: Error finding container 8093ef6feaf726a16768130f9f4cbbe6ff20cda96acf0d1b7145593b0f28ebf8: Status 404 returned error can't find the container with id 8093ef6feaf726a16768130f9f4cbbe6ff20cda96acf0d1b7145593b0f28ebf8
Oct 31 12:38:04.918041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:04.918000    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" event={"ID":"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf","Type":"ContainerStarted","Data":"40ec43d5fd7fa0d2b5cbfd657cdb0eda260eec625a7d9654de79c7d09f5bcc7e"}
Oct 31 12:38:04.918457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:04.918051    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" event={"ID":"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf","Type":"ContainerStarted","Data":"8093ef6feaf726a16768130f9f4cbbe6ff20cda96acf0d1b7145593b0f28ebf8"}
Oct 31 12:38:04.941129 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:04.940072    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podStartSLOduration=1.940028348 podStartE2EDuration="1.940028348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:04.939815546 +0000 UTC m=+609.680241763" watchObservedRunningTime="2026-10-31 12:38:04.940028348 +0000 UTC m=+609.680454568"
Oct 31 12:38:05.065615 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.065578    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:05.830891 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.830857    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"]
Oct 31 12:38:05.835132 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.835111    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.840590 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.840568    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"kube-root-ca.crt\""
Oct 31 12:38:05.840723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.840696    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"simulator-w2-e6c79d-kserve-self-signed-certs\""
Oct 31 12:38:05.840723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.840711    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"default-dockercfg-mscfx\""
Oct 31 12:38:05.840904 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.840886    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"openshift-service-ca.crt\""
Oct 31 12:38:05.845656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.845635    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"]
Oct 31 12:38:05.871240 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.871211    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tmp-dir\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.871445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.871306    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-dshm\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.871445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.871344    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-model-cache\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.871445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.871423    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-home\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.871626 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.871465    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tls-certs\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.871680 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.871594    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzlnn\" (UniqueName: \"kubernetes.io/projected/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-kube-api-access-nzlnn\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.972862 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.972820    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-home\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.973347 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.972897    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tls-certs\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.973347 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.972932    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzlnn\" (UniqueName: \"kubernetes.io/projected/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-kube-api-access-nzlnn\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.973347 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.972991    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tmp-dir\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.973347 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.973084    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-dshm\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.973347 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.973127    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-model-cache\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.973347 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.973246    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-home\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.973347 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.973298    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tmp-dir\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.973775 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.973521    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-model-cache\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.975396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.975345    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-dshm\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.975568 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.975551    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tls-certs\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:05.984884 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:05.984864    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzlnn\" (UniqueName: \"kubernetes.io/projected/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-kube-api-access-nzlnn\") pod \"simulator-w2-e6c79d-kserve-76c67955d5-nvkbq\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") " pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:06.151421 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:06.151322    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:06.431265 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:06.431241    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"]
Oct 31 12:38:06.433562 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:38:06.433535    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cb2a3c0_7255_4be3_93c4_55d9ee673aad.slice/crio-34a1d639aef2698f88caf55a6da7dc1c7233a6034bea28f8e5dd589e95d860c5 WatchSource:0}: Error finding container 34a1d639aef2698f88caf55a6da7dc1c7233a6034bea28f8e5dd589e95d860c5: Status 404 returned error can't find the container with id 34a1d639aef2698f88caf55a6da7dc1c7233a6034bea28f8e5dd589e95d860c5
Oct 31 12:38:06.934842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:06.934756    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" event={"ID":"2cb2a3c0-7255-4be3-93c4-55d9ee673aad","Type":"ContainerStarted","Data":"947c09b8acf76a64abb424d296a32a29eb3351fe41a87bb537162944811522d0"}
Oct 31 12:38:06.934842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:06.934815    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" event={"ID":"2cb2a3c0-7255-4be3-93c4-55d9ee673aad","Type":"ContainerStarted","Data":"34a1d639aef2698f88caf55a6da7dc1c7233a6034bea28f8e5dd589e95d860c5"}
Oct 31 12:38:06.980175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:06.980117    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podStartSLOduration=1.980100238 podStartE2EDuration="1.980100238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:06.979614157 +0000 UTC m=+611.720040377" watchObservedRunningTime="2026-10-31 12:38:06.980100238 +0000 UTC m=+611.720526455"
Oct 31 12:38:07.152316 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:07.152284    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:07.384863 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:07.384815    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:38:07.403442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:07.403413    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:38:07.954217 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:07.954183    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:38:14.065301 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:14.065267    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:14.078245 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:14.078221    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:14.983756 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:14.983724    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:38:16.152061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:16.152017    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:16.165195 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:16.165128    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:16.994822 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:16.994795    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:38:22.475690 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:38:22.475645    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:38:22.852021 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:22.851989    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"]
Oct 31 12:38:22.856010 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:22.855990    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:22.860492 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:22.860467    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-85ce08\"/\"premium-w1-85ce08-kserve-self-signed-certs\""
Oct 31 12:38:22.865699 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:22.865672    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"]
Oct 31 12:38:22.957688 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:22.957650    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tmp-dir\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:22.957886 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:22.957696    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-home\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:22.957886 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:22.957757    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tls-certs\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:22.957886 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:22.957839    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-model-cache\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:22.957886 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:22.957868    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbscw\" (UniqueName: \"kubernetes.io/projected/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-kube-api-access-vbscw\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:22.958114 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:22.957939    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-dshm\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.059095 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.059060    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tmp-dir\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.059095 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.059099    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-home\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.059338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.059128    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tls-certs\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.059338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.059172    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-model-cache\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.059536 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.059508    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tmp-dir\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.059607 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.059546    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-model-cache\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.059607 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.059568    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbscw\" (UniqueName: \"kubernetes.io/projected/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-kube-api-access-vbscw\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.059712 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.059627    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-dshm\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.060153 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.060106    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-home\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.083780 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.080971    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tls-certs\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.085196 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.085161    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-dshm\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.085913 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.085875    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbscw\" (UniqueName: \"kubernetes.io/projected/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-kube-api-access-vbscw\") pod \"premium-w1-85ce08-kserve-8494f7b776-j675m\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") " pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.170450 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.170338    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:23.464748 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:23.464719    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"]
Oct 31 12:38:23.467492 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:38:23.467468    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ae07a54_ebb0_4622_b6e8_8040a947ea0f.slice/crio-42f54a19183be988c1b20722a1dcff2d3c06f64a5e923e12b90082ef8de32b0d WatchSource:0}: Error finding container 42f54a19183be988c1b20722a1dcff2d3c06f64a5e923e12b90082ef8de32b0d: Status 404 returned error can't find the container with id 42f54a19183be988c1b20722a1dcff2d3c06f64a5e923e12b90082ef8de32b0d
Oct 31 12:38:24.020533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:24.020499    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" event={"ID":"4ae07a54-ebb0-4622-b6e8-8040a947ea0f","Type":"ContainerStarted","Data":"a9aff0a174dca2db750479269387a680ae5e8c838e6798eb5a0e6257653f66b0"}
Oct 31 12:38:24.020533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:24.020545    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" event={"ID":"4ae07a54-ebb0-4622-b6e8-8040a947ea0f","Type":"ContainerStarted","Data":"42f54a19183be988c1b20722a1dcff2d3c06f64a5e923e12b90082ef8de32b0d"}
Oct 31 12:38:24.040396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:24.040322    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podStartSLOduration=2.040309915 podStartE2EDuration="2.040309915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:24.038935976 +0000 UTC m=+628.779362193" watchObservedRunningTime="2026-10-31 12:38:24.040309915 +0000 UTC m=+628.780736122"
Oct 31 12:38:24.170904 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:24.170872    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:25.633892 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.633863    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"]
Oct 31 12:38:25.638155 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.638129    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.645709 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.645686    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fb0343\"/\"openshift-service-ca.crt\""
Oct 31 12:38:25.645850 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.645746    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fb0343\"/\"default-dockercfg-5hqp9\""
Oct 31 12:38:25.647269 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.647248    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fb0343\"/\"kube-root-ca.crt\""
Oct 31 12:38:25.647849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.647819    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"]
Oct 31 12:38:25.648143 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.648125    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fb0343\"/\"premium-w0-fb0343-kserve-self-signed-certs\""
Oct 31 12:38:25.786662 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.786628    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-dshm\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.786857 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.786752    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7nd6\" (UniqueName: \"kubernetes.io/projected/e93d0d4e-ec94-4653-9bfb-7dd27581502a-kube-api-access-z7nd6\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.786857 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.786826    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tmp-dir\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.786982 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.786875    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tls-certs\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.786982 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.786945    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-model-cache\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.786982 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.786973    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-home\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.887898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.887823    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7nd6\" (UniqueName: \"kubernetes.io/projected/e93d0d4e-ec94-4653-9bfb-7dd27581502a-kube-api-access-z7nd6\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.887898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.887888    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tmp-dir\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.888124 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.887924    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tls-certs\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.888124 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.887990    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-model-cache\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.888124 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.888023    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-home\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.888124 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.888076    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-dshm\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.888431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.888408    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tmp-dir\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.888507 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.888431    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-model-cache\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.888507 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.888475    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-home\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.890863 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.890839    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tls-certs\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.891257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.891237    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-dshm\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.900673 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.900646    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7nd6\" (UniqueName: \"kubernetes.io/projected/e93d0d4e-ec94-4653-9bfb-7dd27581502a-kube-api-access-z7nd6\") pod \"premium-w0-fb0343-kserve-5cd7b64c58-vg5rm\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") " pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:25.953427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:25.953394    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:26.234001 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:26.233953    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"]
Oct 31 12:38:26.236959 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:38:26.236925    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode93d0d4e_ec94_4653_9bfb_7dd27581502a.slice/crio-530c3f2c90794b7e8ab6a10a3f38bba2d208870a39afb7f24176b79b9ccafc32 WatchSource:0}: Error finding container 530c3f2c90794b7e8ab6a10a3f38bba2d208870a39afb7f24176b79b9ccafc32: Status 404 returned error can't find the container with id 530c3f2c90794b7e8ab6a10a3f38bba2d208870a39afb7f24176b79b9ccafc32
Oct 31 12:38:27.036583 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:27.036550    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" event={"ID":"e93d0d4e-ec94-4653-9bfb-7dd27581502a","Type":"ContainerStarted","Data":"95d49f43d2b23fd6143ff1577ebb53575486efb1961e8ebe999bef55e6e5055d"}
Oct 31 12:38:27.036583 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:27.036589    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" event={"ID":"e93d0d4e-ec94-4653-9bfb-7dd27581502a","Type":"ContainerStarted","Data":"530c3f2c90794b7e8ab6a10a3f38bba2d208870a39afb7f24176b79b9ccafc32"}
Oct 31 12:38:27.056635 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:27.056586    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podStartSLOduration=2.056572955 podStartE2EDuration="2.056572955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:27.055527242 +0000 UTC m=+631.795953461" watchObservedRunningTime="2026-10-31 12:38:27.056572955 +0000 UTC m=+631.796999223"
Oct 31 12:38:27.953579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:27.953543    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:33.170721 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:33.170633    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:33.193485 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:33.193456    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:33.826410 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:33.826377    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"]
Oct 31 12:38:33.832011 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:33.831986    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:33.843342 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:33.840656    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b218d3\"/\"premium-w3-b218d3-kserve-self-signed-certs\""
Oct 31 12:38:33.843342 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:33.843272    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"]
Oct 31 12:38:33.981026 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:33.980982    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa39025-2185-43d3-9e91-2ae5207241c1-tls-certs\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:33.981218 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:33.981083    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzbl9\" (UniqueName: \"kubernetes.io/projected/efa39025-2185-43d3-9e91-2ae5207241c1-kube-api-access-pzbl9\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:33.981218 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:33.981134    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-tmp-dir\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:33.981218 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:33.981160    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-model-cache\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:33.981416 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:33.981234    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-dshm\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:33.981416 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:33.981278    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-home\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.082566 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.082482    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzbl9\" (UniqueName: \"kubernetes.io/projected/efa39025-2185-43d3-9e91-2ae5207241c1-kube-api-access-pzbl9\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.082566 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.082534    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-tmp-dir\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.082566 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.082557    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-model-cache\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.082947 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.082608    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-dshm\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.082947 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.082648    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-home\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.082947 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.082748    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa39025-2185-43d3-9e91-2ae5207241c1-tls-certs\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.083101 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.082969    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-tmp-dir\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.083284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.083249    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-home\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.083450 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.083303    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-model-cache\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.085307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.085285    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-dshm\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.086934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.086909    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa39025-2185-43d3-9e91-2ae5207241c1-tls-certs\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.087495 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.087476    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:38:34.095183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.095157    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzbl9\" (UniqueName: \"kubernetes.io/projected/efa39025-2185-43d3-9e91-2ae5207241c1-kube-api-access-pzbl9\") pod \"premium-w3-b218d3-kserve-5c9776df9-dqtmm\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") " pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.151745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.151714    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:34.426999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:34.426968    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"]
Oct 31 12:38:34.429829 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:38:34.429798    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefa39025_2185_43d3_9e91_2ae5207241c1.slice/crio-5785549ffe3b0ae34c99f6ec5c1ec7d5a483bcbd8e13dd74046901f358e15c93 WatchSource:0}: Error finding container 5785549ffe3b0ae34c99f6ec5c1ec7d5a483bcbd8e13dd74046901f358e15c93: Status 404 returned error can't find the container with id 5785549ffe3b0ae34c99f6ec5c1ec7d5a483bcbd8e13dd74046901f358e15c93
Oct 31 12:38:35.078958 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:35.078919    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" event={"ID":"efa39025-2185-43d3-9e91-2ae5207241c1","Type":"ContainerStarted","Data":"29ef8e25daac3390d513112167cfd937339278b69396214040ff852d40275652"}
Oct 31 12:38:35.079165 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:35.078975    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" event={"ID":"efa39025-2185-43d3-9e91-2ae5207241c1","Type":"ContainerStarted","Data":"5785549ffe3b0ae34c99f6ec5c1ec7d5a483bcbd8e13dd74046901f358e15c93"}
Oct 31 12:38:35.101715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:35.101660    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podStartSLOduration=2.101644415 podStartE2EDuration="2.101644415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:38:35.099735473 +0000 UTC m=+639.840161689" watchObservedRunningTime="2026-10-31 12:38:35.101644415 +0000 UTC m=+639.842070636"
Oct 31 12:38:35.152868 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:35.152839    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:35.954313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:35.954275    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:35.969510 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:35.969483    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:36.095137 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:36.095110    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:38:44.156205 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:44.154993    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:44.185145 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:44.185114    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:45.137403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:38:45.137376    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:38:52.474866 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:38:52.474824    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:39:22.475129 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:39:22.475091    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:39:45.293607 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.293574    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"]
Oct 31 12:39:45.297608 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.297579    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.305418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.305395    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b218d3\"/\"distinct-w3-b218d3-kserve-self-signed-certs\""
Oct 31 12:39:45.315105 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.315078    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"]
Oct 31 12:39:45.490073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.490030    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mgd\" (UniqueName: \"kubernetes.io/projected/113876cb-0de7-4a4c-ae74-9fcdbee19345-kube-api-access-b4mgd\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.490248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.490099    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-tmp-dir\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.490248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.490194    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-dshm\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.490389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.490318    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-home\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.490449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.490399    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113876cb-0de7-4a4c-ae74-9fcdbee19345-tls-certs\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.490548 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.490532    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-model-cache\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.591899 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.591861    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-model-cache\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.592106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.591915    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mgd\" (UniqueName: \"kubernetes.io/projected/113876cb-0de7-4a4c-ae74-9fcdbee19345-kube-api-access-b4mgd\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.592106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.591980    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-tmp-dir\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.592106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.592045    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-dshm\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.592276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.592145    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-home\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.592276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.592172    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113876cb-0de7-4a4c-ae74-9fcdbee19345-tls-certs\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.592399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.592370    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-model-cache\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.592508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.592488    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-tmp-dir\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.592704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.592665    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-home\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.594469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.594440    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-dshm\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.594668 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.594650    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113876cb-0de7-4a4c-ae74-9fcdbee19345-tls-certs\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.607530 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.607506    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mgd\" (UniqueName: \"kubernetes.io/projected/113876cb-0de7-4a4c-ae74-9fcdbee19345-kube-api-access-b4mgd\") pod \"distinct-w3-b218d3-kserve-7bc99557cd-pcpxj\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") " pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.610295 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.610271    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:45.927199 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:45.927096    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"]
Oct 31 12:39:45.928575 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:39:45.928546    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod113876cb_0de7_4a4c_ae74_9fcdbee19345.slice/crio-f1dd99db8c3e61b02267a328acbea76f897e03b12f66cec1286c08ba9100d4ba WatchSource:0}: Error finding container f1dd99db8c3e61b02267a328acbea76f897e03b12f66cec1286c08ba9100d4ba: Status 404 returned error can't find the container with id f1dd99db8c3e61b02267a328acbea76f897e03b12f66cec1286c08ba9100d4ba
Oct 31 12:39:46.465931 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:46.465897    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" event={"ID":"113876cb-0de7-4a4c-ae74-9fcdbee19345","Type":"ContainerStarted","Data":"18f17d48a59e412d374cc66e26d78b03a3a8dd84359981900b8cc601053a9858"}
Oct 31 12:39:46.466320 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:46.465944    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" event={"ID":"113876cb-0de7-4a4c-ae74-9fcdbee19345","Type":"ContainerStarted","Data":"f1dd99db8c3e61b02267a328acbea76f897e03b12f66cec1286c08ba9100d4ba"}
Oct 31 12:39:46.486399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:46.486335    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podStartSLOduration=1.486321631 podStartE2EDuration="1.486321631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:46.485460147 +0000 UTC m=+711.225886364" watchObservedRunningTime="2026-10-31 12:39:46.486321631 +0000 UTC m=+711.226747846"
Oct 31 12:39:46.615643 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:46.615612    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:52.067183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.067145    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"]
Oct 31 12:39:52.071018 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.071001    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.091945 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.091913    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"]
Oct 31 12:39:52.142785 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.142749    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b218d3-openshift-default-dockercfg-kz5nw\""
Oct 31 12:39:52.160403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.160372    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.160533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.160408    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.160533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.160445    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.160605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.160540    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.160605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.160574    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.160671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.160613    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.160801 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.160774    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqxb4\" (UniqueName: \"kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-kube-api-access-vqxb4\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.160962 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.160843    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.160962 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.160905    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.261845 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.261799    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.262044 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.261882    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.262044 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.261935    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqxb4\" (UniqueName: \"kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-kube-api-access-vqxb4\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.262044 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.261983    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.262212 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.262048    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.262212 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.262099    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.262212 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.262128    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.262212 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.262170    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.262433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.262222    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.262679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.262650    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.262783 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.262747    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.263197 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.262957    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.263197 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.263082    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.263197 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.263103    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.264973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.264953    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.265056 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.264978    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.275580 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.275558    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.276573 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.276553    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqxb4\" (UniqueName: \"kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-kube-api-access-vqxb4\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.388494 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.388401    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:52.519971 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:39:52.519927    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:39:52.707722 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.707632    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"]
Oct 31 12:39:52.724837 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.724795    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:52.724948 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.724928    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:52.724990 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:52.724979    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:39:53.510605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.509543    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" event={"ID":"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9","Type":"ContainerStarted","Data":"455b83d46a21d1112ed78e7f574b4c9eb36231c0682284939f1b034e2b637a99"}
Oct 31 12:39:53.510605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.509617    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" event={"ID":"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9","Type":"ContainerStarted","Data":"f725329c7ad125a51439176a6af9f94ef13420f0a6b59c51d7733d0539646f9a"}
Oct 31 12:39:53.544422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.544348    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" podStartSLOduration=1.544334085 podStartE2EDuration="1.544334085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:53.542433529 +0000 UTC m=+718.282859747" watchObservedRunningTime="2026-10-31 12:39:53.544334085 +0000 UTC m=+718.284760300"
Oct 31 12:39:53.827377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.826320    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"]
Oct 31 12:39:53.836456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.833510    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"]
Oct 31 12:39:53.836456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.833621    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:53.840109 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.840069    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b25d3b\"/\"distinct-w5-b25d3b-kserve-self-signed-certs\""
Oct 31 12:39:53.949379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.946835    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-tmp-dir\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:53.949379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.946904    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njvt\" (UniqueName: \"kubernetes.io/projected/e69182cf-2356-4273-84be-13030f3405cd-kube-api-access-4njvt\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:53.949379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.946937    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-dshm\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:53.949379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.946998    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-home\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:53.949379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.947080    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-model-cache\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:53.949379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:53.947508    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69182cf-2356-4273-84be-13030f3405cd-tls-certs\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.049983 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.049930    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-tmp-dir\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.050778 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.050751    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njvt\" (UniqueName: \"kubernetes.io/projected/e69182cf-2356-4273-84be-13030f3405cd-kube-api-access-4njvt\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.051320 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.051297    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-dshm\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.053435 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.053407    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-home\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.053942 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.053925    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-model-cache\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.054148 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.054133    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69182cf-2356-4273-84be-13030f3405cd-tls-certs\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.061194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.057669    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-model-cache\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.061194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.053809    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-home\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.061194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.050666    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-tmp-dir\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.076148 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.076085    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njvt\" (UniqueName: \"kubernetes.io/projected/e69182cf-2356-4273-84be-13030f3405cd-kube-api-access-4njvt\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.076949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.076896    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69182cf-2356-4273-84be-13030f3405cd-tls-certs\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.091212 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.090509    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-ms7fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:39:54.091212 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.090585    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:39:54.092470 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.092414    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-dshm\") pod \"distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") " pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.156347 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.156316    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:54.407575 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.407427    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:39:54.408830 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.408494    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 12:39:54.408830 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.408548    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" podUID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 12:39:54.524382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:54.524333    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"]
Oct 31 12:39:54.527980 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:39:54.527951    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode69182cf_2356_4273_84be_13030f3405cd.slice/crio-3df5ef2fbd331c218626f6036feee6db192997b0f185848c709322711898ed8f WatchSource:0}: Error finding container 3df5ef2fbd331c218626f6036feee6db192997b0f185848c709322711898ed8f: Status 404 returned error can't find the container with id 3df5ef2fbd331c218626f6036feee6db192997b0f185848c709322711898ed8f
Oct 31 12:39:55.400007 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:55.399752    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 12:39:55.400007 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:55.399812    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" podUID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 12:39:55.521096 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:55.521057    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" event={"ID":"e69182cf-2356-4273-84be-13030f3405cd","Type":"ContainerStarted","Data":"c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a"}
Oct 31 12:39:55.521255 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:55.521105    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" event={"ID":"e69182cf-2356-4273-84be-13030f3405cd","Type":"ContainerStarted","Data":"3df5ef2fbd331c218626f6036feee6db192997b0f185848c709322711898ed8f"}
Oct 31 12:39:55.549707 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:55.549655    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podStartSLOduration=2.549641231 podStartE2EDuration="2.549641231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:39:55.548036125 +0000 UTC m=+720.288462340" watchObservedRunningTime="2026-10-31 12:39:55.549641231 +0000 UTC m=+720.290067476"
Oct 31 12:39:55.610874 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:55.610838    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:55.624959 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:55.624931    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:56.162376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:56.161432    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:39:56.389874 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:56.389840    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 12:39:56.390052 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:56.389908    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" podUID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 12:39:56.537449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:56.537365    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:39:57.389496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:57.389453    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 12:39:57.389910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:57.389530    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" podUID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 12:39:58.389259 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:58.389218    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 12:39:58.389473 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:58.389273    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" podUID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 12:39:59.389819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:59.389776    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 12:39:59.390325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:39:59.389836    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" podUID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 12:40:00.616207 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:00.616175    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:40:00.616704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:00.616614    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:40:00.617584 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:00.617558    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:40:04.157280 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:04.157200    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:40:04.170103 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:04.170078    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:40:04.585658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:04.585628    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:40:07.252044 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.252009    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"]
Oct 31 12:40:07.255725 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.255704    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.267551 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.267531    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c67de2-openshift-default-dockercfg-psh8p\""
Oct 31 12:40:07.282629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.282603    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"]
Oct 31 12:40:07.362106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.362070    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"]
Oct 31 12:40:07.365945 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.365920    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.372770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.372740    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-85ce08-openshift-default-dockercfg-5nn7z\""
Oct 31 12:40:07.384956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.384917    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"]
Oct 31 12:40:07.399897 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.399863    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5tjx\" (UniqueName: \"kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-kube-api-access-q5tjx\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.400040 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.399901    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.400040 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.399930    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37639489-380a-4eab-b2c0-f1a61362d2db-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.400127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.400057    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37639489-380a-4eab-b2c0-f1a61362d2db-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.400127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.400101    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.400243 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.400160    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.400243 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.400208    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.400311 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.400247    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.400369 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.400343    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502080    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502140    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502173    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502244    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kst48\" (UniqueName: \"kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-kube-api-access-kst48\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502275    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502310    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502335    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502389    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5tjx\" (UniqueName: \"kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-kube-api-access-q5tjx\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502427    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502452    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502480    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502515    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37639489-380a-4eab-b2c0-f1a61362d2db-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502539    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502584    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502616    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37639489-380a-4eab-b2c0-f1a61362d2db-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502657    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502713    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.502798    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.503280    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.503611    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.504426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.503917    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.506704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.505917    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.507423 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.507114    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37639489-380a-4eab-b2c0-f1a61362d2db-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.510259 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.510214    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37639489-380a-4eab-b2c0-f1a61362d2db-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.511254 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.511219    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.516858 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.516833    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5tjx\" (UniqueName: \"kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-kube-api-access-q5tjx\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.518953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.518931    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.569533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.569490    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:07.604134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.604095    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.604134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.604136    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.604386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.604202    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kst48\" (UniqueName: \"kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-kube-api-access-kst48\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.604386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.604240    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.604386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.604314    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.604572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.604427    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.604572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.604452    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.604572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.604491    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.604572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.604498    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.604572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.604550    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.604868 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.604843    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.604931 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.604907    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.605124 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.605098    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.605124 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.605100    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.606703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.606679    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.607445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.607424    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.617974 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.617946    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.618378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.618332    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kst48\" (UniqueName: \"kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-kube-api-access-kst48\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.680813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.680777    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:07.855823 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.855798    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"]
Oct 31 12:40:07.857692 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:40:07.857664    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37639489_380a_4eab_b2c0_f1a61362d2db.slice/crio-be5b520c83a4cee3ce41c35aac4686794904eca44cdd602691954741418741c2 WatchSource:0}: Error finding container be5b520c83a4cee3ce41c35aac4686794904eca44cdd602691954741418741c2: Status 404 returned error can't find the container with id be5b520c83a4cee3ce41c35aac4686794904eca44cdd602691954741418741c2
Oct 31 12:40:07.860037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.860004    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:07.860135 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.860091    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:07.860135 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.860123    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:07.975751 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.975715    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"]
Oct 31 12:40:07.990181 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:40:07.990146    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78f4e51_551c_43f8_a199_01ebd2c11e1a.slice/crio-94ab1f1bdb7fadc65983cf7a2fc59cdca9ecd5072c4abe8b6bd9b39d70d36304 WatchSource:0}: Error finding container 94ab1f1bdb7fadc65983cf7a2fc59cdca9ecd5072c4abe8b6bd9b39d70d36304: Status 404 returned error can't find the container with id 94ab1f1bdb7fadc65983cf7a2fc59cdca9ecd5072c4abe8b6bd9b39d70d36304
Oct 31 12:40:07.993338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.993297    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:07.993474 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.993398    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:07.993474 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:07.993440    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:08.646152 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:08.646113    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" event={"ID":"a78f4e51-551c-43f8-a199-01ebd2c11e1a","Type":"ContainerStarted","Data":"74475ac74a756ceaad112d68e691f6cfa356dca69a75f9978308ac3d0fffa7e4"}
Oct 31 12:40:08.646668 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:08.646213    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" event={"ID":"a78f4e51-551c-43f8-a199-01ebd2c11e1a","Type":"ContainerStarted","Data":"94ab1f1bdb7fadc65983cf7a2fc59cdca9ecd5072c4abe8b6bd9b39d70d36304"}
Oct 31 12:40:08.647766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:08.647741    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" event={"ID":"37639489-380a-4eab-b2c0-f1a61362d2db","Type":"ContainerStarted","Data":"8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c"}
Oct 31 12:40:08.647912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:08.647899    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" event={"ID":"37639489-380a-4eab-b2c0-f1a61362d2db","Type":"ContainerStarted","Data":"be5b520c83a4cee3ce41c35aac4686794904eca44cdd602691954741418741c2"}
Oct 31 12:40:08.676955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:08.676892    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" podStartSLOduration=1.676874926 podStartE2EDuration="1.676874926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:08.673842904 +0000 UTC m=+733.414269126" watchObservedRunningTime="2026-10-31 12:40:08.676874926 +0000 UTC m=+733.417301145"
Oct 31 12:40:08.683390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:08.683331    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:09.577387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:09.576706    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:09.598377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:09.597501    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 12:40:09.598377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:09.597572    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 12:40:09.684376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:09.684299    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 31 12:40:09.684872 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:09.684392    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 31 12:40:09.988433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:09.986444    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:10.133374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:10.132442    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:40:10.571379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:10.570741    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 12:40:10.571379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:10.570804    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 12:40:10.575382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:10.575197    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:10.683842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:10.683808    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 31 12:40:10.684095 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:10.684058    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 31 12:40:10.879563 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:10.878786    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:10.978376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:10.977439    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:40:11.087991 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.087945    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:40:11.134552 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.134447    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:40:11.135471 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.135449    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:40:11.163825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.163760    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" podStartSLOduration=4.1637363 podStartE2EDuration="4.1637363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:08.702706171 +0000 UTC m=+733.443132400" watchObservedRunningTime="2026-10-31 12:40:11.1637363 +0000 UTC m=+735.904162518"
Oct 31 12:40:11.568378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.566815    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:40:11.577393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.576635    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 12:40:11.577393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.576703    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 12:40:11.625890 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.625844    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:11.698380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.669746    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:40:11.720522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.720483    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 31 12:40:11.724448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.724412    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 31 12:40:11.885539 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.881090    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:11.885539 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.883440    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:40:11.977420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.941303    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:11.978261 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.978226    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:40:11.978747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.978727    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:40:12.014376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:11.990319    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:12.077495 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:12.077442    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:12.580374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:12.573758    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 12:40:12.580374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:12.573817    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 12:40:12.580374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:12.579124    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:40:12.672378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:12.672192    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:40:12.691279 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:12.691242    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 31 12:40:12.691572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:12.691544    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 31 12:40:12.810066 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:12.809755    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:40:12.938475 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:12.909264    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:40:12.981037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:12.980992    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:13.140419 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:13.138342    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:40:13.253062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:13.252977    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:40:13.292405 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:13.292363    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-lsjpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:40:13.292703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:13.292665    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx" podUID="6cacf32b-0e2d-48cc-a0be-c99e26159bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:13.572070 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:13.572031    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 12:40:13.572403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:13.572341    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 12:40:13.688778 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:13.688741    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 31 12:40:13.689014 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:13.688989    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 31 12:40:14.567378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.559204    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"]
Oct 31 12:40:14.575373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.573703    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 12:40:14.575373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.573757    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 12:40:14.625777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.625739    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"]
Oct 31 12:40:14.626051 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.625860    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.636658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.636630    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.636850 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.636832    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.637019 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.637000    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.637182 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.637164    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ad591e-3b45-4312-8612-cf848dcf55e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.637327 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.637311    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.637483 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.637465    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpk57\" (UniqueName: \"kubernetes.io/projected/57ad591e-3b45-4312-8612-cf848dcf55e5-kube-api-access-dpk57\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.637626 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.637610    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.637753 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.637737    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ad591e-3b45-4312-8612-cf848dcf55e5-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.637880 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.637864    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ad591e-3b45-4312-8612-cf848dcf55e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.684377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.683690    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 31 12:40:14.684377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.683751    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 31 12:40:14.738808 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.738773    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.738957 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.738815    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpk57\" (UniqueName: \"kubernetes.io/projected/57ad591e-3b45-4312-8612-cf848dcf55e5-kube-api-access-dpk57\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.738957 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.738861    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.738957 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.738881    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ad591e-3b45-4312-8612-cf848dcf55e5-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.738957 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.738900    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ad591e-3b45-4312-8612-cf848dcf55e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.739165 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.738976    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.739165 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.739140    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.739276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.739219    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.739276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.739272    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.739422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.739322    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ad591e-3b45-4312-8612-cf848dcf55e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.739486 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.739472    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.739593 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.739543    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ad591e-3b45-4312-8612-cf848dcf55e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.739811 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.739792    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.739869 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.739813    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.741739 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.741717    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ad591e-3b45-4312-8612-cf848dcf55e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.741891 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.741871    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ad591e-3b45-4312-8612-cf848dcf55e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.750983 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.750921    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpk57\" (UniqueName: \"kubernetes.io/projected/57ad591e-3b45-4312-8612-cf848dcf55e5-kube-api-access-dpk57\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.751784 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.751761    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ad591e-3b45-4312-8612-cf848dcf55e5-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-dswld\" (UID: \"57ad591e-3b45-4312-8612-cf848dcf55e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:14.943051 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:14.942973    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:15.296787 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:15.296759    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"]
Oct 31 12:40:15.299561 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:40:15.299529    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ad591e_3b45_4312_8612_cf848dcf55e5.slice/crio-141f559d60364c48fc81b13711ae6ed2a589550406e749eaf4138430a8f4d400 WatchSource:0}: Error finding container 141f559d60364c48fc81b13711ae6ed2a589550406e749eaf4138430a8f4d400: Status 404 returned error can't find the container with id 141f559d60364c48fc81b13711ae6ed2a589550406e749eaf4138430a8f4d400
Oct 31 12:40:15.302633 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:15.302598    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:15.302768 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:15.302669    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:15.302768 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:15.302707    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:15.570193 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:15.570143    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 12:40:15.570714 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:15.570210    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 12:40:15.917672 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:15.917642    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" event={"ID":"57ad591e-3b45-4312-8612-cf848dcf55e5","Type":"ContainerStarted","Data":"b92a669b98bdf2c9717ee3dbf7c4de1d7d4af2c5ab76520ae95d363572a88891"}
Oct 31 12:40:15.917856 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:15.917843    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" event={"ID":"57ad591e-3b45-4312-8612-cf848dcf55e5","Type":"ContainerStarted","Data":"141f559d60364c48fc81b13711ae6ed2a589550406e749eaf4138430a8f4d400"}
Oct 31 12:40:15.949384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:15.943749    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:15.949384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:15.946493    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" podStartSLOduration=1.946477458 podStartE2EDuration="1.946477458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:15.942400368 +0000 UTC m=+740.682826610" watchObservedRunningTime="2026-10-31 12:40:15.946477458 +0000 UTC m=+740.686903675"
Oct 31 12:40:16.724539 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:16.724491    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:40:16.725068 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:16.724554    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:16.940852 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:16.940815    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:16.941732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:16.941682    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:16.942534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:16.942512    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:40:16.944884 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:16.944864    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-dswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 12:40:16.944933 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:16.944908    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" podUID="57ad591e-3b45-4312-8612-cf848dcf55e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 12:40:17.302610 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:17.302584    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:17.303314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:17.303275    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:17.304142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:17.304127    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 12:40:17.943367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:17.943327    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-dswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 12:40:17.943845 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:17.943389    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" podUID="57ad591e-3b45-4312-8612-cf848dcf55e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 12:40:18.944079 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:18.944034    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-dswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 12:40:18.944482 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:18.944108    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" podUID="57ad591e-3b45-4312-8612-cf848dcf55e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 12:40:19.943540 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:19.943503    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-dswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 12:40:19.943719 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:19.943575    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" podUID="57ad591e-3b45-4312-8612-cf848dcf55e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 12:40:20.943411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:20.943374    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-dswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 12:40:20.943909 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:20.943465    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" podUID="57ad591e-3b45-4312-8612-cf848dcf55e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 12:40:21.946677 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:21.946642    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-dswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 12:40:21.947320 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:21.947288    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" podUID="57ad591e-3b45-4312-8612-cf848dcf55e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 12:40:22.467484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.467449    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"]
Oct 31 12:40:22.484402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.475927    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.504465 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.503390    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e6c79d-openshift-default-dockercfg-hwsts\""
Oct 31 12:40:22.509769 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.509737    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"]
Oct 31 12:40:22.535153 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:40:22.535119    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:40:22.567292 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.567247    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.567463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.567298    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.567529 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.567499    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.567572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.567550    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x78r2\" (UniqueName: \"kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-kube-api-access-x78r2\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.567669 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.567651    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.567720 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.567705    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.567862 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.567846    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1778cc89-0d95-46b0-b633-66a43483dc2d-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.567928 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.567915    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.567969 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.567952    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.669209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.669170    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.669432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.669256    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.669432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.669280    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78r2\" (UniqueName: \"kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-kube-api-access-x78r2\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.669432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.669307    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.669432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.669381    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.669658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.669446    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1778cc89-0d95-46b0-b633-66a43483dc2d-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.669658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.669500    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.669658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.669531    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.669658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.669638    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.669885 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.669725    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.669949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.669929    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.670070 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.670014    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.670153 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.670099    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.670278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.670256    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1778cc89-0d95-46b0-b633-66a43483dc2d-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.671755 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.671727    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.671926 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.671902    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.683754 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.683733    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x78r2\" (UniqueName: \"kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-kube-api-access-x78r2\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.683860 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.683836    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.824787 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.824757    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:22.945049 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.944978    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-dswld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 12:40:22.945049 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:22.945040    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" podUID="57ad591e-3b45-4312-8612-cf848dcf55e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 12:40:23.119168 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.119140    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"]
Oct 31 12:40:23.122164 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:40:23.122137    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1778cc89_0d95_46b0_b633_66a43483dc2d.slice/crio-b91d3cc010b70bad169c3be8055e77e2c93bf7a098525a8e65caa94109165b31 WatchSource:0}: Error finding container b91d3cc010b70bad169c3be8055e77e2c93bf7a098525a8e65caa94109165b31: Status 404 returned error can't find the container with id b91d3cc010b70bad169c3be8055e77e2c93bf7a098525a8e65caa94109165b31
Oct 31 12:40:23.124645 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.124607    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:23.124745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.124689    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:23.124745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.124736    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:23.343376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.342766    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"]
Oct 31 12:40:23.350400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.348683    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.354861 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.354834    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e6c79d\"/\"distinct-w2-e6c79d-kserve-self-signed-certs\""
Oct 31 12:40:23.356851 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.356823    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"]
Oct 31 12:40:23.377154 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.377084    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-model-cache\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.377154 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.377149    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118349cf-bd83-48fb-860a-42ec40ab2285-tls-certs\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.377456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.377201    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvrmd\" (UniqueName: \"kubernetes.io/projected/118349cf-bd83-48fb-860a-42ec40ab2285-kube-api-access-wvrmd\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.377456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.377230    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-home\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.377456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.377270    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-dshm\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.377456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.377295    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-tmp-dir\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.483379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.478796    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-model-cache\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.483379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.478864    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118349cf-bd83-48fb-860a-42ec40ab2285-tls-certs\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.483379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.478912    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvrmd\" (UniqueName: \"kubernetes.io/projected/118349cf-bd83-48fb-860a-42ec40ab2285-kube-api-access-wvrmd\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.483379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.478942    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-home\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.483379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.478984    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-dshm\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.483379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.479011    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-tmp-dir\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.483379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.479630    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-tmp-dir\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.483379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.479865    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-home\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.483379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.480198    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-model-cache\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.483379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.482709    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-dshm\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.483379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.483088    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118349cf-bd83-48fb-860a-42ec40ab2285-tls-certs\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.523157 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.523081    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvrmd\" (UniqueName: \"kubernetes.io/projected/118349cf-bd83-48fb-860a-42ec40ab2285-kube-api-access-wvrmd\") pod \"distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") " pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:23.683555 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:23.676560    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:24.034770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:24.034679    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55" event={"ID":"1778cc89-0d95-46b0-b633-66a43483dc2d","Type":"ContainerStarted","Data":"b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329"}
Oct 31 12:40:24.034917 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:24.034784    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55" event={"ID":"1778cc89-0d95-46b0-b633-66a43483dc2d","Type":"ContainerStarted","Data":"b91d3cc010b70bad169c3be8055e77e2c93bf7a098525a8e65caa94109165b31"}
Oct 31 12:40:24.049780 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:24.049752    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"]
Oct 31 12:40:24.051663 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:40:24.051632    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod118349cf_bd83_48fb_860a_42ec40ab2285.slice/crio-db374a13aab2302b32d5c9abcdb0afb50bb36c76e5cf749d0425ee1961d312d2 WatchSource:0}: Error finding container db374a13aab2302b32d5c9abcdb0afb50bb36c76e5cf749d0425ee1961d312d2: Status 404 returned error can't find the container with id db374a13aab2302b32d5c9abcdb0afb50bb36c76e5cf749d0425ee1961d312d2
Oct 31 12:40:24.095007 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:24.094939    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55" podStartSLOduration=2.094923458 podStartE2EDuration="2.094923458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:24.093898272 +0000 UTC m=+748.834324489" watchObservedRunningTime="2026-10-31 12:40:24.094923458 +0000 UTC m=+748.835349678"
Oct 31 12:40:24.809697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:24.809666    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:24.810172 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:24.809973    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:24.810857 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:24.810836    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld"
Oct 31 12:40:24.825152 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:24.825127    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:24.826836 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:24.826808    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 12:40:24.826974 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:24.826863    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55" podUID="1778cc89-0d95-46b0-b633-66a43483dc2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 12:40:25.047584 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:25.047551    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" event={"ID":"118349cf-bd83-48fb-860a-42ec40ab2285","Type":"ContainerStarted","Data":"e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5"}
Oct 31 12:40:25.047781 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:25.047599    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" event={"ID":"118349cf-bd83-48fb-860a-42ec40ab2285","Type":"ContainerStarted","Data":"db374a13aab2302b32d5c9abcdb0afb50bb36c76e5cf749d0425ee1961d312d2"}
Oct 31 12:40:25.078425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:25.076956    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podStartSLOduration=2.076935223 podStartE2EDuration="2.076935223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:25.07569346 +0000 UTC m=+749.816119677" watchObservedRunningTime="2026-10-31 12:40:25.076935223 +0000 UTC m=+749.817361420"
Oct 31 12:40:25.683484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:25.683453    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:25.825705 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:25.825665    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 12:40:25.826205 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:25.825720    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55" podUID="1778cc89-0d95-46b0-b633-66a43483dc2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 12:40:26.826060 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:26.826023    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 12:40:26.826596 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:26.826075    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55" podUID="1778cc89-0d95-46b0-b633-66a43483dc2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 12:40:27.825920 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:27.825886    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 12:40:27.826129 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:27.825940    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55" podUID="1778cc89-0d95-46b0-b633-66a43483dc2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 12:40:28.826042 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:28.826003    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 12:40:28.826252 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:28.826070    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55" podUID="1778cc89-0d95-46b0-b633-66a43483dc2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 12:40:30.165978 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:30.165944    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:30.167278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:30.167251    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:30.168137 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:30.168107    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:40:33.677369 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:33.677323    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:33.691370 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:33.691333    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:34.176544 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:34.176509    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:40:35.320078 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.320043    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"]
Oct 31 12:40:35.325436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.325408    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.334536 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.334508    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fb0343\"/\"distinct-2-w0-fb0343-kserve-self-signed-certs\""
Oct 31 12:40:35.335543 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.335517    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"]
Oct 31 12:40:35.390145 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.390113    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-dshm\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.390316 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.390161    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tmp-dir\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.390393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.390307    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-model-cache\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.390393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.390375    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-home\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.390490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.390467    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tls-certs\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.390545 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.390526    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47x8n\" (UniqueName: \"kubernetes.io/projected/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-kube-api-access-47x8n\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.497060 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.497023    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-dshm\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.497823 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.497788    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tmp-dir\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.498142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.498123    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-model-cache\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.498428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.498335    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-home\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.499969 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.499951    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tls-certs\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.500175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.500149    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47x8n\" (UniqueName: \"kubernetes.io/projected/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-kube-api-access-47x8n\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.512545 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.499889    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-home\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.512798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.508965    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-dshm\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.512958 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.499123    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tmp-dir\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.513070 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.499661    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-model-cache\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.517295 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.517263    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tls-certs\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.541777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.541279    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47x8n\" (UniqueName: \"kubernetes.io/projected/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-kube-api-access-47x8n\") pod \"distinct-2-w0-fb0343-kserve-58487fb754-sfbw2\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") " pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.643720 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.643620    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:35.939213 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:35.939191    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"]
Oct 31 12:40:35.941137 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:40:35.941113    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eddddf5_9ee1_4d43_a506_ac57824bc0c4.slice/crio-90d69d906eaa99231b50293e7cc1f67ac5a62e79851aab6210591b85e3740f04 WatchSource:0}: Error finding container 90d69d906eaa99231b50293e7cc1f67ac5a62e79851aab6210591b85e3740f04: Status 404 returned error can't find the container with id 90d69d906eaa99231b50293e7cc1f67ac5a62e79851aab6210591b85e3740f04
Oct 31 12:40:36.179317 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.179284    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" event={"ID":"4eddddf5-9ee1-4d43-a506-ac57824bc0c4","Type":"ContainerStarted","Data":"79e58cb93e2407e0ac9f2c853560649c88d9c89f96f9303951dace223049a109"}
Oct 31 12:40:36.179543 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.179329    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" event={"ID":"4eddddf5-9ee1-4d43-a506-ac57824bc0c4","Type":"ContainerStarted","Data":"90d69d906eaa99231b50293e7cc1f67ac5a62e79851aab6210591b85e3740f04"}
Oct 31 12:40:36.200981 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.200889    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podStartSLOduration=1.200874687 podStartE2EDuration="1.200874687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:36.199135181 +0000 UTC m=+760.939561399" watchObservedRunningTime="2026-10-31 12:40:36.200874687 +0000 UTC m=+760.941300903"
Oct 31 12:40:36.644671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.644631    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:36.805397 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.805363    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"]
Oct 31 12:40:36.809202 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.809181    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:36.814020 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.813996    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fb0343-openshift-default-dockercfg-pktb6\""
Oct 31 12:40:36.822859 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.822834    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"]
Oct 31 12:40:36.934708 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.934630    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:36.934858 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.934748    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:36.934858 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.934780    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:36.934936 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.934883    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:36.934999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.934979    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:36.935041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.935017    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:36.935041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.935035    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9h4z\" (UniqueName: \"kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-kube-api-access-f9h4z\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:36.935260 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.935237    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:36.935347 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:36.935277    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.036588 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.036551    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.036588 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.036590    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.036811 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.036612    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.036811 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.036657    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.036811 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.036672    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.036811 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.036712    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.036811 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.036764    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.037119 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.036814    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.037119 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.036876    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9h4z\" (UniqueName: \"kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-kube-api-access-f9h4z\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.037119 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.037035    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.037119 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.037066    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.037314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.037142    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.037516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.037485    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.037603 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.037592    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.039475 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.039451    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.039611 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.039539    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.049298 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.049264    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9h4z\" (UniqueName: \"kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-kube-api-access-f9h4z\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.049470 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.049449    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.126714 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.126679    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:37.415838 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.415810    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"]
Oct 31 12:40:37.417799 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:40:37.417774    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2c3768_6d11_40a9_85a9_92aca25c3e6c.slice/crio-e1de06b80e2fd11b9c3b04efdd9b48329ce62e6d5376f3c3b1012d4d416ef4dd WatchSource:0}: Error finding container e1de06b80e2fd11b9c3b04efdd9b48329ce62e6d5376f3c3b1012d4d416ef4dd: Status 404 returned error can't find the container with id e1de06b80e2fd11b9c3b04efdd9b48329ce62e6d5376f3c3b1012d4d416ef4dd
Oct 31 12:40:37.420235 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.420197    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:37.420341 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.420267    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:37.420341 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:37.420298    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:38.194686 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:38.194645    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms" event={"ID":"ef2c3768-6d11-40a9-85a9-92aca25c3e6c","Type":"ContainerStarted","Data":"66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419"}
Oct 31 12:40:38.195067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:38.194702    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms" event={"ID":"ef2c3768-6d11-40a9-85a9-92aca25c3e6c","Type":"ContainerStarted","Data":"e1de06b80e2fd11b9c3b04efdd9b48329ce62e6d5376f3c3b1012d4d416ef4dd"}
Oct 31 12:40:38.223906 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:38.223851    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms" podStartSLOduration=2.22383774 podStartE2EDuration="2.22383774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:38.221714084 +0000 UTC m=+762.962140313" watchObservedRunningTime="2026-10-31 12:40:38.22383774 +0000 UTC m=+762.964263956"
Oct 31 12:40:39.126978 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:39.126943    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:39.128756 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:39.128729    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 12:40:39.128879 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:39.128779    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms" podUID="ef2c3768-6d11-40a9-85a9-92aca25c3e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 12:40:40.127733 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:40.127693    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 12:40:40.128143 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:40.127750    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms" podUID="ef2c3768-6d11-40a9-85a9-92aca25c3e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 12:40:41.127221 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:41.127181    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 12:40:41.127432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:41.127234    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms" podUID="ef2c3768-6d11-40a9-85a9-92aca25c3e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 12:40:42.127179 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:42.127147    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 12:40:42.127580 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:42.127208    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms" podUID="ef2c3768-6d11-40a9-85a9-92aca25c3e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 12:40:43.127869 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:43.127833    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 12:40:43.128273 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:43.127900    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms" podUID="ef2c3768-6d11-40a9-85a9-92aca25c3e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 12:40:44.127465 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:44.127423    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 12:40:44.127739 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:44.127494    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms" podUID="ef2c3768-6d11-40a9-85a9-92aca25c3e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 12:40:45.132597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:45.132565    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:45.133082 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:45.132856    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:45.133609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:45.133588    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:40:45.644575 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:45.644539    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:45.657793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:45.657766    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:46.252865 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:46.252837    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:40:51.987379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:51.987323    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"]
Oct 31 12:40:52.030537 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.030508    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.037274 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.037198    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-67d410-openshift-default-dockercfg-p2p5s\""
Oct 31 12:40:52.044393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.038311    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"]
Oct 31 12:40:52.156244 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.156212    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.156452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.156276    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.156452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.156310    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.156452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.156416    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.156680 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.156484    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.156680 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.156595    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.156806 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.156786    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljzfz\" (UniqueName: \"kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-kube-api-access-ljzfz\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.156944 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.156834    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.156944 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.156869    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.257931 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.257839    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.257931 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.257889    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.257931 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.257914    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.258153 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.257964    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.258153 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.258046    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.258153 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.258081    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.258262 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.258217    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljzfz\" (UniqueName: \"kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-kube-api-access-ljzfz\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.258300 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.258269    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.258339 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.258308    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.259449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.259343    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.260048 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.260024    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.260429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.260347    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.261853 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.261822    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.262093 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.262072    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.263161 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.263136    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.263989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.263619    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.272828 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.272809    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.275973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.275951    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljzfz\" (UniqueName: \"kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-kube-api-access-ljzfz\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.431605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.431573    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:52.475517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.475480    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"]
Oct 31 12:40:52.480445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.480423    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.509520 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.495489    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"]
Oct 31 12:40:52.509520 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:40:52.499341    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:40:52.561506 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.561457    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.561661 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.561513    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.561661 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.561612    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.561661 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.561650    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8kh\" (UniqueName: \"kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-kube-api-access-nt8kh\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.562075 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.561763    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.562075 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.561967    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.562075 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.561996    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.562075 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.562044    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.562390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.562082    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.663182 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.663129    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.663381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.663196    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8kh\" (UniqueName: \"kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-kube-api-access-nt8kh\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.663381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.663248    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.663381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.663305    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.663381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.663332    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.663620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.663418    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.663620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.663479    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.663620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.663514    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.663620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.663547    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.663883 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.663738    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.664454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.664073    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.664454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.664077    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.664454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.664150    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.664454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.664420    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.665949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.665927    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.666086 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.666069    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.675760 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.675738    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8kh\" (UniqueName: \"kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-kube-api-access-nt8kh\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.676151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.676130    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:52.740282 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.740258    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"]
Oct 31 12:40:52.742169 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:40:52.742143    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb163c8ec_22bc_4195_a75b_a9e5602c69ad.slice/crio-f98a18b5f6eccf744917821404643a9ca7a86ee2aa0a2d1b1d56a3e669b875a3 WatchSource:0}: Error finding container f98a18b5f6eccf744917821404643a9ca7a86ee2aa0a2d1b1d56a3e669b875a3: Status 404 returned error can't find the container with id f98a18b5f6eccf744917821404643a9ca7a86ee2aa0a2d1b1d56a3e669b875a3
Oct 31 12:40:52.744719 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.744686    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:52.744787 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.744751    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:52.744787 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.744779    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:52.813148 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:52.813124    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:53.112782 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:53.111777    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"]
Oct 31 12:40:53.117638 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:40:53.117611    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c999a6_2915_4a7b_8f8f_2862bf0f880e.slice/crio-7d55b8ea81502b2b91314713a15cc03d8d60f172fb118de8c60fed3925a60857 WatchSource:0}: Error finding container 7d55b8ea81502b2b91314713a15cc03d8d60f172fb118de8c60fed3925a60857: Status 404 returned error can't find the container with id 7d55b8ea81502b2b91314713a15cc03d8d60f172fb118de8c60fed3925a60857
Oct 31 12:40:53.120168 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:53.120127    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:53.120258 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:53.120219    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:53.120304 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:53.120262    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:40:53.318650 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:53.318616    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" event={"ID":"d0c999a6-2915-4a7b-8f8f-2862bf0f880e","Type":"ContainerStarted","Data":"7d55b8ea81502b2b91314713a15cc03d8d60f172fb118de8c60fed3925a60857"}
Oct 31 12:40:53.320778 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:53.320745    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" event={"ID":"b163c8ec-22bc-4195-a75b-a9e5602c69ad","Type":"ContainerStarted","Data":"8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7"}
Oct 31 12:40:53.320893 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:53.320801    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" event={"ID":"b163c8ec-22bc-4195-a75b-a9e5602c69ad","Type":"ContainerStarted","Data":"f98a18b5f6eccf744917821404643a9ca7a86ee2aa0a2d1b1d56a3e669b875a3"}
Oct 31 12:40:53.372427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:53.367054    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" podStartSLOduration=2.367031122 podStartE2EDuration="2.367031122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:53.351430715 +0000 UTC m=+778.091856932" watchObservedRunningTime="2026-10-31 12:40:53.367031122 +0000 UTC m=+778.107457338"
Oct 31 12:40:53.435757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:53.435720    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:40:53.447921 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:53.446497    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 12:40:53.447921 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:53.446559    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 12:40:54.355959 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:54.355921    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" event={"ID":"d0c999a6-2915-4a7b-8f8f-2862bf0f880e","Type":"ContainerStarted","Data":"e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372"}
Oct 31 12:40:54.386796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:54.386651    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" podStartSLOduration=2.386630012 podStartE2EDuration="2.386630012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:40:54.384860771 +0000 UTC m=+779.125286990" watchObservedRunningTime="2026-10-31 12:40:54.386630012 +0000 UTC m=+779.127056230"
Oct 31 12:40:54.432151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:54.432117    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 12:40:54.432426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:54.432397    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 12:40:54.823631 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:54.818103    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:40:54.823631 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:54.820505    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 12:40:54.823631 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:54.820568    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 12:40:55.177438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:55.175446    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:55.207377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:55.206520    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:55.464436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:55.463681    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 12:40:55.464436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:55.463743    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 12:40:55.577375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:55.574900    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:40:55.727437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:55.726457    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:40:55.813865 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:55.813828    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 12:40:55.814131 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:55.814106    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 12:40:56.435269 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:56.435221    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 12:40:56.435562 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:56.435536    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 12:40:56.814111 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:56.814044    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 12:40:56.814693 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:56.814159    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 12:40:57.432619 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:57.432579    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 12:40:57.432824 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:57.432653    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 12:40:57.818245 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:57.817778    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 12:40:57.818245 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:57.817844    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 12:40:58.432093 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:58.432043    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 12:40:58.432306 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:58.432118    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 12:40:58.821378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:58.818800    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 12:40:58.821378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:58.818861    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 12:40:59.433549 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:59.433516    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 12:40:59.433807 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:59.433778    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 12:40:59.815905 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:59.815870    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 12:40:59.816232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:40:59.816188    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 12:41:00.432438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:00.432395    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 12:41:00.432836 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:00.432476    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 12:41:00.814294 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:00.814250    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 12:41:00.814519 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:00.814329    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 12:41:01.440445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:01.440416    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:41:01.440861 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:01.440842    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:41:01.441610 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:01.441585    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 12:41:01.817851 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:01.817825    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:41:01.818139 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:01.818119    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:41:01.818762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:01.818747    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:41:07.237864 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.237826    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"]
Oct 31 12:41:07.241533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.241512    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.265144 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.265116    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"]
Oct 31 12:41:07.270233 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.270205    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"]
Oct 31 12:41:07.292914 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.292883    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.295365 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.295316    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"]
Oct 31 12:41:07.339819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.339784    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65535656-2606-4bda-b022-52ad8dd432b2-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.339989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.339842    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69sq7\" (UniqueName: \"kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-kube-api-access-69sq7\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.339989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.339941    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.339989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.339973    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.340134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.339993    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.340134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340090    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.340239 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340148    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da085219-1b5e-4976-9f9b-45e98b491499-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.340239 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340184    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxsb\" (UniqueName: \"kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-kube-api-access-jcxsb\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.340391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340249    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.340391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340291    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da085219-1b5e-4976-9f9b-45e98b491499-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.340514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340430    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.340514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340485    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.340609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340555    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.340656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340598    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65535656-2606-4bda-b022-52ad8dd432b2-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.340711 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340681    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.340791 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340777    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.340864 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340845    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.340900 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.340890    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.441526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.441488    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.441526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.441527    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.441788 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.441569    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.441788 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.441595    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65535656-2606-4bda-b022-52ad8dd432b2-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.441916 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.441895    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.441984 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.441935    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69sq7\" (UniqueName: \"kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-kube-api-access-69sq7\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.442054 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442006    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.442054 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442023    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.442176 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442082    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.442176 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442149    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.442295 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442186    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.442295 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442230    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da085219-1b5e-4976-9f9b-45e98b491499-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.442295 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442259    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxsb\" (UniqueName: \"kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-kube-api-access-jcxsb\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.442504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442335    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.442504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442341    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.442504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442416    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da085219-1b5e-4976-9f9b-45e98b491499-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.442504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442452    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65535656-2606-4bda-b022-52ad8dd432b2-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.442504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442483    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.442745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442528    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.442745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442575    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.442745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442598    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65535656-2606-4bda-b022-52ad8dd432b2-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.442745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442633    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.442745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.442677    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.443028 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.443013    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-credential-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.443485 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.443278    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-socket\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.443631 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.443599    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-certs\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.443631 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.443620    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da085219-1b5e-4976-9f9b-45e98b491499-istiod-ca-cert\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.443967 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.443930    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-data\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.445090 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.445061    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.445468 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.445446    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da085219-1b5e-4976-9f9b-45e98b491499-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.445876 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.445850    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-envoy\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.446108 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.446090    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65535656-2606-4bda-b022-52ad8dd432b2-istio-podinfo\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.454099 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.454067    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.454205 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.454069    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-istio-token\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.454712 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.454691    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxsb\" (UniqueName: \"kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-kube-api-access-jcxsb\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.454783 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.454693    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69sq7\" (UniqueName: \"kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-kube-api-access-69sq7\") pod \"e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") " pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.555909 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.555882    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:07.609818 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.609695    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:07.734178 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.734138    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"]
Oct 31 12:41:07.739425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.739400    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.797715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.793588    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.797715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.793724    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.797715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.793756    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.797715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.793796    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.797715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.793854    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.797715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.793915    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.797715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.793990    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.797715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.794073    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wjx\" (UniqueName: \"kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-kube-api-access-d7wjx\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.797715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.794103    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5f61f8-fc82-4778-a23e-5326e73fb615-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.801643 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.801548    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b25d3b-openshift-default-dockercfg-5r5zv\""
Oct 31 12:41:07.809398 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.809307    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"]
Oct 31 12:41:07.856281 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.856221    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"]
Oct 31 12:41:07.867837 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:41:07.867811    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65535656_2606_4bda_b022_52ad8dd432b2.slice/crio-367182ec309822b936bc1b66443f6632b893fe8257c4bd66dabf39ba1e0a0702 WatchSource:0}: Error finding container 367182ec309822b936bc1b66443f6632b893fe8257c4bd66dabf39ba1e0a0702: Status 404 returned error can't find the container with id 367182ec309822b936bc1b66443f6632b893fe8257c4bd66dabf39ba1e0a0702
Oct 31 12:41:07.870773 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.870742    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:07.870866 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.870822    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:07.870947 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.870867    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:07.894882 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.894841    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.894940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.894895    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.895066 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.895022    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.895164 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.895110    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.895215 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.895178    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.895257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.895246    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.895314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.895279    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.895383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.895339    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wjx\" (UniqueName: \"kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-kube-api-access-d7wjx\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.895433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.895386    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5f61f8-fc82-4778-a23e-5326e73fb615-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.895476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.895442    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.895875 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.895852    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.895963 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.895898    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.895963 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.895938    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.896459 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.896437    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5f61f8-fc82-4778-a23e-5326e73fb615-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.897666 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.897644    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.897894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.897876    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.908128 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.908104    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wjx\" (UniqueName: \"kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-kube-api-access-d7wjx\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.922124 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.922099    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"]
Oct 31 12:41:07.924203 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.924182    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:07.930631 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:41:07.930607    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda085219_1b5e_4976_9f9b_45e98b491499.slice/crio-0ff7c74d556969e155c3bd43b8279f309564f305938afeea21033b6fe6931b93 WatchSource:0}: Error finding container 0ff7c74d556969e155c3bd43b8279f309564f305938afeea21033b6fe6931b93: Status 404 returned error can't find the container with id 0ff7c74d556969e155c3bd43b8279f309564f305938afeea21033b6fe6931b93
Oct 31 12:41:07.933442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.933411    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:07.933507 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.933475    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:07.933507 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:07.933503    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:08.125297 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.125206    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:08.445644 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.445548    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"]
Oct 31 12:41:08.489380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.487919    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:08.489380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.488014    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:08.489380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.488053    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:08.528799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.528764    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" event={"ID":"6e5f61f8-fc82-4778-a23e-5326e73fb615","Type":"ContainerStarted","Data":"64cfd10743b1dc72b77aedf7dcab05b693c16f2520a5c8ecac8ad488a7ca5f95"}
Oct 31 12:41:08.530645 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.530371    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" event={"ID":"65535656-2606-4bda-b022-52ad8dd432b2","Type":"ContainerStarted","Data":"8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f"}
Oct 31 12:41:08.530645 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.530425    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" event={"ID":"65535656-2606-4bda-b022-52ad8dd432b2","Type":"ContainerStarted","Data":"367182ec309822b936bc1b66443f6632b893fe8257c4bd66dabf39ba1e0a0702"}
Oct 31 12:41:08.532631 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.532602    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" event={"ID":"da085219-1b5e-4976-9f9b-45e98b491499","Type":"ContainerStarted","Data":"f599ed87ea81f98d02499ad7cff2ff4eb75587917bbe6d7bcdd815cd4fbf41cb"}
Oct 31 12:41:08.532767 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.532654    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" event={"ID":"da085219-1b5e-4976-9f9b-45e98b491499","Type":"ContainerStarted","Data":"0ff7c74d556969e155c3bd43b8279f309564f305938afeea21033b6fe6931b93"}
Oct 31 12:41:08.563392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.560712    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:08.563801 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.563770    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 12:41:08.564283 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.563965    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 12:41:08.570082 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.569944    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" podStartSLOduration=1.5699287389999999 podStartE2EDuration="1.569928739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:08.569732197 +0000 UTC m=+793.310158414" watchObservedRunningTime="2026-10-31 12:41:08.569928739 +0000 UTC m=+793.310354956"
Oct 31 12:41:08.610877 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.610838    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:08.612953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.612648    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 12:41:08.612953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:08.612702    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 12:41:09.548017 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:09.547977    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" event={"ID":"6e5f61f8-fc82-4778-a23e-5326e73fb615","Type":"ContainerStarted","Data":"d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83"}
Oct 31 12:41:09.556253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:09.556213    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 12:41:09.556448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:09.556266    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 12:41:09.576870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:09.576778    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" podStartSLOduration=2.576755079 podStartE2EDuration="2.576755079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:08.610111577 +0000 UTC m=+793.350537801" watchObservedRunningTime="2026-10-31 12:41:09.576755079 +0000 UTC m=+794.317181300"
Oct 31 12:41:09.584386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:09.582656    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" podStartSLOduration=2.582640309 podStartE2EDuration="2.582640309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:09.581799699 +0000 UTC m=+794.322225915" watchObservedRunningTime="2026-10-31 12:41:09.582640309 +0000 UTC m=+794.323066526"
Oct 31 12:41:09.611403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:09.610798    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 12:41:09.611403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:09.610861    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 12:41:10.133378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:10.128824    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:10.133378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:10.128974    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:41:10.133378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:10.129019    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" podUID="6e5f61f8-fc82-4778-a23e-5326e73fb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 12:41:10.577378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:10.568723    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 12:41:10.577378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:10.568785    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 12:41:10.619325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:10.618541    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 12:41:10.619325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:10.618600    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 12:41:11.135381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:11.129769    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:41:11.135381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:11.129850    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" podUID="6e5f61f8-fc82-4778-a23e-5326e73fb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 12:41:11.560434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:11.560397    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 12:41:11.568804 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:11.568413    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 12:41:11.613890 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:11.613855    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 12:41:11.620227 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:11.620178    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 12:41:12.129380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:12.126771    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:41:12.129380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:12.126840    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" podUID="6e5f61f8-fc82-4778-a23e-5326e73fb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 12:41:12.575374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:12.566766    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 12:41:12.575374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:12.566831    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 12:41:12.613930 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:12.613892    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 12:41:12.614664 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:12.614622    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 12:41:13.126764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:13.126720    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:41:13.126983 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:13.126787    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" podUID="6e5f61f8-fc82-4778-a23e-5326e73fb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 12:41:13.576048 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:13.576011    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 12:41:13.576375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:13.576318    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 12:41:13.612920 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:13.612887    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 12:41:13.613152 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:13.613129    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 12:41:14.131771 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:14.131734    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:41:14.132420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:14.132392    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" podUID="6e5f61f8-fc82-4778-a23e-5326e73fb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 12:41:14.559815 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:14.559773    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 12:41:14.560692 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:14.560250    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 12:41:14.611378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:14.610811    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 12:41:14.611378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:14.610866    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 12:41:15.104411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.103690    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"]
Oct 31 12:41:15.113451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.113423    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.122177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.120664    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-67d410\"/\"distinct-2-w6-67d410-kserve-self-signed-certs\""
Oct 31 12:41:15.125785 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.125760    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:41:15.125978 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.125954    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" podUID="6e5f61f8-fc82-4778-a23e-5326e73fb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 12:41:15.137674 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.137648    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"]
Oct 31 12:41:15.219433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.219394    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-745w8\" (UniqueName: \"kubernetes.io/projected/f0b4371b-f550-4a21-8c82-f2d84f836b67-kube-api-access-745w8\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.219738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.219697    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-model-cache\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.219885 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.219870    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-tmp-dir\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.220084 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.220061    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-home\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.220283 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.220265    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-dshm\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.220710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.220668    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b4371b-f550-4a21-8c82-f2d84f836b67-tls-certs\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.322508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.322425    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-home\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.323725 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.323703    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-dshm\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.324424 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.324402    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b4371b-f550-4a21-8c82-f2d84f836b67-tls-certs\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.332373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.325026    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-745w8\" (UniqueName: \"kubernetes.io/projected/f0b4371b-f550-4a21-8c82-f2d84f836b67-kube-api-access-745w8\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.332925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.332900    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-model-cache\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.333087 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.333064    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-tmp-dir\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.333401 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.323580    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-home\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.333739 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.333717    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-tmp-dir\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.334118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.334070    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-model-cache\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.337908 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.337883    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-dshm\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.340513 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.340491    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b4371b-f550-4a21-8c82-f2d84f836b67-tls-certs\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.375271 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.375194    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-745w8\" (UniqueName: \"kubernetes.io/projected/f0b4371b-f550-4a21-8c82-f2d84f836b67-kube-api-access-745w8\") pod \"distinct-2-w6-67d410-kserve-7748c5c886-xbqwq\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") " pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.477375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.468299    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:15.595387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.590752    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 12:41:15.595387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.590835    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 12:41:15.654587 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.622738    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 12:41:15.654587 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.622795    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 12:41:15.997431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:15.997397    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"]
Oct 31 12:41:16.006309 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:41:16.006274    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0b4371b_f550_4a21_8c82_f2d84f836b67.slice/crio-aa232ee078a20e2d394158d4baec1254bfb4d8fe3d3fc6b52c0f4c6c6a16e8dd WatchSource:0}: Error finding container aa232ee078a20e2d394158d4baec1254bfb4d8fe3d3fc6b52c0f4c6c6a16e8dd: Status 404 returned error can't find the container with id aa232ee078a20e2d394158d4baec1254bfb4d8fe3d3fc6b52c0f4c6c6a16e8dd
Oct 31 12:41:16.126090 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:16.126050    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 12:41:16.126177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:16.126108    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" podUID="6e5f61f8-fc82-4778-a23e-5326e73fb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 12:41:16.587717 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:16.587683    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:16.590276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:16.590252    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:16.607678 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:16.607652    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:41:16.641694 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:16.641653    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:16.642248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:16.642228    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:16.643623 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:16.643604    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:41:16.673179 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:16.673144    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" event={"ID":"f0b4371b-f550-4a21-8c82-f2d84f836b67","Type":"ContainerStarted","Data":"a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918"}
Oct 31 12:41:16.673378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:16.673189    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" event={"ID":"f0b4371b-f550-4a21-8c82-f2d84f836b67","Type":"ContainerStarted","Data":"aa232ee078a20e2d394158d4baec1254bfb4d8fe3d3fc6b52c0f4c6c6a16e8dd"}
Oct 31 12:41:16.704667 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:16.702077    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podStartSLOduration=1.702058114 podStartE2EDuration="1.702058114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:16.697752352 +0000 UTC m=+801.438178570" watchObservedRunningTime="2026-10-31 12:41:16.702058114 +0000 UTC m=+801.442484332"
Oct 31 12:41:17.167367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:17.167318    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:17.168596 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:17.168524    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:17.169483 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:17.169462    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:41:17.469108 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:17.469034    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:20.910507 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:20.910460    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:41:20.944490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:20.944444    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:20.985575 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:20.985516    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:21.078543 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.078487    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:21.087783 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.087730    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:41:21.144251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.143217    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:41:21.211624 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.211518    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:21.250377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.249435    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:21.486434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.483451    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:21.584385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.582477    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:41:21.683377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.679484    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:21.915376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.914852    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:21.915920 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.915887    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:41:21.927377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.919425    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"]
Oct 31 12:41:21.937384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.935963    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:21.955115 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.953150    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:41:21.955115 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.953733    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:41:22.009395 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.986051    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:22.009395 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:21.986764    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:41:22.117466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.087183    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.117466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.087250    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.117466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.087330    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.117466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.087416    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.117466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.087542    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.117466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.087586    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhwwz\" (UniqueName: \"kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-kube-api-access-hhwwz\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.117466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.087771    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.117466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.087887    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.117466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.087977    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.117466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.090061    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:41:22.121959 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.121935    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:41:22.132061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.123021    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:22.132061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.123410    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:41:22.154057 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.153982    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:22.154450 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.154426    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:41:22.187980 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.187531    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:41:22.192819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.192728    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.192964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.192862    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.192964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.192924    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.193086 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.192962    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.193086 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.193016    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.193197 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.193134    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.193197 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.193178    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhwwz\" (UniqueName: \"kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-kube-api-access-hhwwz\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.193303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.193238    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.193303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.193279    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.219797 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.219346    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:22.264376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.251873    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.270793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.269724    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.293214 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.285883    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:22.293214 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.290911    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.293214 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.291594    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.293214 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.291858    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.293214 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.291938    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:41:22.293214 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.291995    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:41:22.293214 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.292032    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:41:22.321390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.314850    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.321390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.319513    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:41:22.321390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.319573    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:41:22.326315 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.326286    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:41:22.332893 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.332867    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"]
Oct 31 12:41:22.395117 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.395029    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.442439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.429071    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhwwz\" (UniqueName: \"kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-kube-api-access-hhwwz\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.445992 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.445927    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.705425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.704711    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:22.786377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.785298    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:41:22.827455 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.827427    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:41:22.841771 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.840804    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:41:22.880128 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:41:22.880055    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:41:22.926005 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:22.925976    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:41:23.221366 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:23.221320    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"]
Oct 31 12:41:23.223052 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:41:23.223027    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39d5190_02f9_4b13_be8a_c6d4f8f2b3a1.slice/crio-5ea5f0e5559ab9ece4a4e640688267ccc3ec2807d2b49e36d58c59a95299e501 WatchSource:0}: Error finding container 5ea5f0e5559ab9ece4a4e640688267ccc3ec2807d2b49e36d58c59a95299e501: Status 404 returned error can't find the container with id 5ea5f0e5559ab9ece4a4e640688267ccc3ec2807d2b49e36d58c59a95299e501
Oct 31 12:41:23.225672 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:23.225638    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:23.225744 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:23.225702    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:23.225744 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:23.225730    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:23.858701 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:23.857483    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps" event={"ID":"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1","Type":"ContainerStarted","Data":"11c826778ba2532b5801dff6a58fbf665ff1905994dae760878abdcd00c7faeb"}
Oct 31 12:41:23.858701 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:23.857596    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps" event={"ID":"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1","Type":"ContainerStarted","Data":"5ea5f0e5559ab9ece4a4e640688267ccc3ec2807d2b49e36d58c59a95299e501"}
Oct 31 12:41:23.894126 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:23.894074    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps" podStartSLOduration=2.894057551 podStartE2EDuration="2.894057551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:23.891777028 +0000 UTC m=+808.632203257" watchObservedRunningTime="2026-10-31 12:41:23.894057551 +0000 UTC m=+808.634484103"
Oct 31 12:41:24.706589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:24.706555    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:24.708481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:24.708454    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 12:41:24.708588 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:24.708506    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps" podUID="a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 12:41:25.477854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:25.477819    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:25.493888 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:25.493860    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:25.706452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:25.706417    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 12:41:25.706617 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:25.706469    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps" podUID="a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 12:41:25.988597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:25.988568    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:26.706380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:26.706313    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 12:41:26.706568 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:26.706403    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps" podUID="a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 12:41:27.706406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:27.706344    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 12:41:27.706806 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:27.706423    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps" podUID="a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 12:41:28.707144 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:28.707108    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 12:41:28.707557 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:28.707171    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps" podUID="a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 12:41:29.708929 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:29.708891    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 12:41:29.709550 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:29.709515    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps" podUID="a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 12:41:30.961067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:30.961039    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:30.961602 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:30.961428    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:30.962878 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:30.962856    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:41:37.461844 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.461762    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"]
Oct 31 12:41:37.465991 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.465962    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.481736 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.481709    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab46e85-e4a5-4b1e-a334-1dad88fde563-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.481933 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.481782    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjsx8\" (UniqueName: \"kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-kube-api-access-kjsx8\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.482296 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.482265    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.482462 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.482313    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.482576 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.482553    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.482658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.482622    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.482658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.482653    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.482777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.482714    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.482777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.482761    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.510697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.510660    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"]
Oct 31 12:41:37.526879 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.526843    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"]
Oct 31 12:41:37.530958 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.530866    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.562962 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.562927    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"]
Oct 31 12:41:37.585129 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585097    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.585303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585143    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.585303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585196    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.585303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585241    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.585303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585273    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.585303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585296    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.585766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585374    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.585766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585400    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.585766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585451    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.585766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585523    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab46e85-e4a5-4b1e-a334-1dad88fde563-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.585766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585565    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsx8\" (UniqueName: \"kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-kube-api-access-kjsx8\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.585766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585619    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.585766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585648    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.585766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585679    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.585766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585732    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.585766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585756    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfbhm\" (UniqueName: \"kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-kube-api-access-mfbhm\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.586488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.585868    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.586551 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.586513    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.586993 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.586960    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.587740 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.587715    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab46e85-e4a5-4b1e-a334-1dad88fde563-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.588287 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.588262    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.589704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.589669    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.591741 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.591717    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.592003 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.591982    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.595252 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.594897    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.602726 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.602696    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.602995 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.602973    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjsx8\" (UniqueName: \"kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-kube-api-access-kjsx8\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.641816 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.641777    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"]
Oct 31 12:41:37.647843 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.647820    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.675854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.675815    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"]
Oct 31 12:41:37.688326 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688294    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.688491 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688347    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.688491 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688396    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnw7g\" (UniqueName: \"kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-kube-api-access-tnw7g\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.688491 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688458    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.688652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688557    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.688652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688597    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.688724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688666    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.688724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688697    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.688825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688751    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.688825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688776    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.688825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688790    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.688825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688810    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.689027 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688893    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.689027 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688947    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.689027 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.688971    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfbhm\" (UniqueName: \"kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-kube-api-access-mfbhm\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.689027 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.689015    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.689237 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.689045    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-credential-socket\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.689237 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.689050    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.689237 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.689098    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.689237 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.689100    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-data\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.689237 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.689168    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.689546 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.689388    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-certs\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.690189 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.690161    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istiod-ca-cert\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.691963 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.691933    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-podinfo\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.692081 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.692023    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-envoy\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.716767 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.716664    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfbhm\" (UniqueName: \"kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-kube-api-access-mfbhm\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.718998 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.718968    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-token\") pod \"e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") " pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.719669 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.719646    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"]
Oct 31 12:41:37.724751 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.724730    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.742918 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.742881    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"]
Oct 31 12:41:37.790268 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790228    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.790472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790279    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9780b010-8c66-4490-8f1a-82296b85235e-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.790472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790313    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.790472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790343    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.790472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790433    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.790643 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790528    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.790643 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790555    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.790643 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790582    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.790643 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790606    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rf22\" (UniqueName: \"kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-kube-api-access-8rf22\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.790830 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790647    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnw7g\" (UniqueName: \"kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-kube-api-access-tnw7g\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.790830 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790686    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9780b010-8c66-4490-8f1a-82296b85235e-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.790830 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790719    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.790830 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790747    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.790830 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790756    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-certs\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.790830 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790763    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.791121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790888    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.791121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790933    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.791121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.790994    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-credential-socket\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.791121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.791032    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.791121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.791052    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istiod-ca-cert\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.791344 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.791190    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.791344 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.791294    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.791630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.791608    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-data\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.793242 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.793214    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-envoy\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.793555 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.793534    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-podinfo\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.803952 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.803928    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnw7g\" (UniqueName: \"kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-kube-api-access-tnw7g\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.804082 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.804007    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-token\") pod \"e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") " pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:37.818119 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.818094    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:37.856880 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.856841    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:37.893070 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.893041    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.893229 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.893111    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rf22\" (UniqueName: \"kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-kube-api-access-8rf22\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.893229 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.893182    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9780b010-8c66-4490-8f1a-82296b85235e-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.893388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.893320    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.893459 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.893409    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.893513 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.893487    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.893569 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.893554    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9780b010-8c66-4490-8f1a-82296b85235e-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.893628 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.893592    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.893628 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.893621    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.894403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.894326    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.894600 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.894581    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.895300 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.895276    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9780b010-8c66-4490-8f1a-82296b85235e-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.895534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.895515    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.896278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.896255    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.898561 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.898518    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9780b010-8c66-4490-8f1a-82296b85235e-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.899939 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.899858    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.906903 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.906842    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rf22\" (UniqueName: \"kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-kube-api-access-8rf22\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.908048 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.908024    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:37.976995 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:37.976896    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:38.038797 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.038758    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:38.120395 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.120365    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"]
Oct 31 12:41:38.129098 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:41:38.129074    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab46e85_e4a5_4b1e_a334_1dad88fde563.slice/crio-e0de129525cb9696706d98b51da4ea1c9ab09e173d5797745d1af95fc5c2d2e4 WatchSource:0}: Error finding container e0de129525cb9696706d98b51da4ea1c9ab09e173d5797745d1af95fc5c2d2e4: Status 404 returned error can't find the container with id e0de129525cb9696706d98b51da4ea1c9ab09e173d5797745d1af95fc5c2d2e4
Oct 31 12:41:38.159153 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.147643    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.159153 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.148098    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.159153 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.148433    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.196721 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.196690    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"]
Oct 31 12:41:38.217578 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:41:38.217549    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a25fb34_1e28_435d_8ff9_430f2719f1f2.slice/crio-6375b6df72f27b88c8eb02d88321b7ebb388f1a5119cba135aeccc0877963020 WatchSource:0}: Error finding container 6375b6df72f27b88c8eb02d88321b7ebb388f1a5119cba135aeccc0877963020: Status 404 returned error can't find the container with id 6375b6df72f27b88c8eb02d88321b7ebb388f1a5119cba135aeccc0877963020
Oct 31 12:41:38.221324 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.221289    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.221407 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.221374    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.221452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.221422    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.274852 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.274810    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"]
Oct 31 12:41:38.278599 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:41:38.278557    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf969552_deff_4db0_91e7_45ff2b8a9c2b.slice/crio-59f91e20b6a26b7c1feaeb5710a0d13d8b7a798146b09ce058642bc8f95e4126 WatchSource:0}: Error finding container 59f91e20b6a26b7c1feaeb5710a0d13d8b7a798146b09ce058642bc8f95e4126: Status 404 returned error can't find the container with id 59f91e20b6a26b7c1feaeb5710a0d13d8b7a798146b09ce058642bc8f95e4126
Oct 31 12:41:38.284954 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.284221    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.284954 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.284309    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.284954 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.284365    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.415191 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.414775    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"]
Oct 31 12:41:38.418991 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:41:38.418947    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9780b010_8c66_4490_8f1a_82296b85235e.slice/crio-613fa47f07f6829860be78659f2ae71ca922db1acf93941f63897107fc420159 WatchSource:0}: Error finding container 613fa47f07f6829860be78659f2ae71ca922db1acf93941f63897107fc420159: Status 404 returned error can't find the container with id 613fa47f07f6829860be78659f2ae71ca922db1acf93941f63897107fc420159
Oct 31 12:41:38.421783 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.421751    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.421968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.421825    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:38.421968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:38.421854    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:41:39.170377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.169199    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" event={"ID":"cf969552-deff-4db0-91e7-45ff2b8a9c2b","Type":"ContainerStarted","Data":"ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3"}
Oct 31 12:41:39.170377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.169292    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" event={"ID":"cf969552-deff-4db0-91e7-45ff2b8a9c2b","Type":"ContainerStarted","Data":"59f91e20b6a26b7c1feaeb5710a0d13d8b7a798146b09ce058642bc8f95e4126"}
Oct 31 12:41:39.185377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.183160    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" event={"ID":"0a25fb34-1e28-435d-8ff9-430f2719f1f2","Type":"ContainerStarted","Data":"53c61f7118bb57ae0d445b4498bf657a6312f38a4aa3fe159400db630dac93d5"}
Oct 31 12:41:39.185377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.183226    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" event={"ID":"0a25fb34-1e28-435d-8ff9-430f2719f1f2","Type":"ContainerStarted","Data":"6375b6df72f27b88c8eb02d88321b7ebb388f1a5119cba135aeccc0877963020"}
Oct 31 12:41:39.190375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.186525    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" event={"ID":"eab46e85-e4a5-4b1e-a334-1dad88fde563","Type":"ContainerStarted","Data":"53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1"}
Oct 31 12:41:39.190375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.186590    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" event={"ID":"eab46e85-e4a5-4b1e-a334-1dad88fde563","Type":"ContainerStarted","Data":"e0de129525cb9696706d98b51da4ea1c9ab09e173d5797745d1af95fc5c2d2e4"}
Oct 31 12:41:39.201201 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.201161    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" event={"ID":"9780b010-8c66-4490-8f1a-82296b85235e","Type":"ContainerStarted","Data":"0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83"}
Oct 31 12:41:39.201531 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.201500    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" event={"ID":"9780b010-8c66-4490-8f1a-82296b85235e","Type":"ContainerStarted","Data":"613fa47f07f6829860be78659f2ae71ca922db1acf93941f63897107fc420159"}
Oct 31 12:41:39.326954 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.323337    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podStartSLOduration=2.323315154 podStartE2EDuration="2.323315154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:39.235305413 +0000 UTC m=+823.975731654" watchObservedRunningTime="2026-10-31 12:41:39.323315154 +0000 UTC m=+824.063741373"
Oct 31 12:41:39.404391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.386756    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" podStartSLOduration=2.3867343 podStartE2EDuration="2.3867343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:39.36086181 +0000 UTC m=+824.101288027" watchObservedRunningTime="2026-10-31 12:41:39.3867343 +0000 UTC m=+824.127160520"
Oct 31 12:41:39.420451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.413182    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" podStartSLOduration=2.413166562 podStartE2EDuration="2.413166562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:39.401513706 +0000 UTC m=+824.141939917" watchObservedRunningTime="2026-10-31 12:41:39.413166562 +0000 UTC m=+824.153592778"
Oct 31 12:41:39.461177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.461080    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" podStartSLOduration=2.461058789 podStartE2EDuration="2.461058789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:41:39.459187979 +0000 UTC m=+824.199614198" watchObservedRunningTime="2026-10-31 12:41:39.461058789 +0000 UTC m=+824.201485006"
Oct 31 12:41:39.823404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.821761    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:39.823404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.823301    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 12:41:39.823687 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.823365    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 12:41:39.867385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.864138    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:39.867385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.864273    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:41:39.867385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.864334    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 12:41:39.981390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.979588    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:39.981390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.979720    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:41:39.981390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:39.979765    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 12:41:40.041391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.041177    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:40.041391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.041373    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:41:40.041659 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.041424    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" podUID="9780b010-8c66-4490-8f1a-82296b85235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 12:41:40.194459 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.193458    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:40.272380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.254368    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:40.273117 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.272719    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:40.368379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.365441    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:41:40.586892 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.586847    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:40.794382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.793824    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:40.819375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.818723    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 12:41:40.819375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.818789    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 12:41:40.851571 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.851477    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:41:40.857919 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.857889    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:41:40.858136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.858110    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 12:41:40.994714 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.973496    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:40.995587 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:40.995273    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:41:41.115880 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.115530    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:41:41.121387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.116511    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 12:41:41.121387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.116585    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" podUID="9780b010-8c66-4490-8f1a-82296b85235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 12:41:41.281496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.281451    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:41.302377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.284032    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:41:41.302377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.282196    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:41:41.302377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.285626    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:41:41.341432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.326448    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:41.341432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.327047    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:41:41.341432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.329208    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:41.406433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.404434    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:41.406433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.404860    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:41:41.477378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.447375    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:41:41.758475 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.754450    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:41.758475 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.756431    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:41:41.819386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.813730    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:41.873380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.859973    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 12:41:41.873380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.860035    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 12:41:41.873380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.861442    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:41.873380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.868757    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:41.947101 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.947071    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:41:41.947400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.947381    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:41:41.947620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.947605    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:41:41.949734 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.949710    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:41:41.949910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.949889    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 12:41:41.951430 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.951411    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:41:41.952106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.952082    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:42.000381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.985894    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:41:42.000381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.985990    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 12:41:42.000381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:41.995855    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:41:42.023492 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.021654    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:42.103378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.102774    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:41:42.103378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.102887    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" podUID="9780b010-8c66-4490-8f1a-82296b85235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 12:41:42.546380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.544005    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:41:42.570702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.550373    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:42.570702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.551074    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:42.570702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.553848    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:41:42.607007 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.606958    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:42.614785 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.614677    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:42.623990 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.622789    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:41:42.623990 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.622856    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:41:42.656422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.656396    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:41:42.744377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.743977    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:42.759414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.759369    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:41:42.911376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.904010    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:41:42.911376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.904387    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:41:42.911376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.905256    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:41:42.950392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.950341    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:41:42.982442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.950669    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 12:41:42.982757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.953705    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 12:41:42.982960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.982915    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 12:41:42.983125 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.962053    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:42.984765 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.983858    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:41:42.995394 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.991458    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:41:42.995394 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:42.991504    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 12:41:43.079788 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.079746    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:41:43.080088 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.080051    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" podUID="9780b010-8c66-4490-8f1a-82296b85235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 12:41:43.268250 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.268155    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:41:43.303035 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.303011    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:41:43.589175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.571672    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:43.647988 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.640585    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:43.684375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.683335    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:43.810391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.784187    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Oct 31 12:41:43.815380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.815323    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:43.832679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.828902    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 12:41:43.832679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.828959    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 12:41:43.902465 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.894374    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:41:43.902465 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.894447    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 12:41:43.932842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.921848    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:41:43.932842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.922206    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:41:43.985655 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.985619    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.003507 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.003462    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 12:41:44.004129 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:43.986090    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:44.055427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.050148    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.055427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.050201    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" podUID="9780b010-8c66-4490-8f1a-82296b85235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 12:41:44.080343 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.079389    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:41:44.257438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.221947    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:41:44.304377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.300532    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:44.579253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.578640    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:41:44.580696 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.578154    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:41:44.677633 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.677578    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 31 12:41:44.749381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.711579    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:41:44.749381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.712282    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:41:44.788378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.761445    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:44.810400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.808584    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:41:44.820390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.818830    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.820390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.818889    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 12:41:44.866001 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.865906    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:41:44.866280 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.866232    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 12:41:44.924476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.924431    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.010439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:44.982069    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:41:45.010859 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.010830    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 12:41:45.011564 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.010761    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.013467 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.013437    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.052427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.052328    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:41:45.052719 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.052677    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" podUID="9780b010-8c66-4490-8f1a-82296b85235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 12:41:45.083752 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.083707    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.161540 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.161444    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.316376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.315542    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.331536 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.316611    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:41:45.582550 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.582506    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.583207 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.583177    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.714565 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.714521    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:41:45.776432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.762668    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.776432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.763101    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:45.821380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.818034    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:45.821380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.819492    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 12:41:45.821380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.819536    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 12:41:45.876027 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.875987    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:41:45.921651 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.921594    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 12:41:45.931515 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.931473    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:46.022736 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:45.972192    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:41:46.065476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.002894    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:41:46.065476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.014444    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:46.065476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.042592    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:41:46.065476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.043267    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" podUID="9780b010-8c66-4490-8f1a-82296b85235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 12:41:46.065476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.042781    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 12:41:46.065476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.014533    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Oct 31 12:41:46.065476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.014481    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 12:41:46.065476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.043737    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:41:46.236729 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.236683    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:41:46.349553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.349451    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:41:46.478381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.475056    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 31 12:41:46.478381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.477381    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:41:46.541733 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.540206    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:41:46.554370 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.549446    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:41:46.554676 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.554634    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:46.566912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.566889    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:41:46.590408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.586420    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:46.611881 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.611797    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 31 12:41:46.612188 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.612159    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:46.612679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.612657    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:41:46.659538 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.659510    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:41:46.752508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.750626    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:46.764490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.763809    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:46.773463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.771870    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:41:46.789373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.788154    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:41:46.828857 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.828814    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 12:41:46.829043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.828870    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 12:41:46.859203 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.859169    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 12:41:46.859501 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.859472    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 12:41:46.911020 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.910704    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:41:46.911318 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.911297    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:47.000064 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.999707    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:41:47.000064 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:46.999764    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 12:41:47.039783 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:47.039756    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 12:41:47.040059 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:47.040030    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" podUID="9780b010-8c66-4490-8f1a-82296b85235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 12:41:47.880255 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:47.880225    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:47.886483 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:47.886460    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:47.901833 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:47.901806    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 12:41:47.978449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:47.978410    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 12:41:47.978643 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:47.978473    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 12:41:48.467298 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:48.467265    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:48.467793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:48.467727    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:48.469218 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:48.469194    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:41:48.733463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:48.733342    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:48.736642 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:48.734257    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:48.736642 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:48.736522    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:41:49.009325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:49.008011    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:49.009325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:49.009027    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:49.014481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:49.014456    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:41:52.884820 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:41:52.884765    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:41:53.231443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:53.229448    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:53.372636 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:53.372473    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:41:53.596249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:53.596199    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:41:53.705848 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:53.619564    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:53.707671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:53.658516    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:53.716287 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:53.677656    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:53.717148 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:53.677779    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:41:53.813991 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:53.813952    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:41:53.814972 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:53.814939    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:53.815444 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:53.815422    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:41:53.835705 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:53.835669    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:41:53.836261 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:53.836232    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Oct 31 12:41:54.015832 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.015001    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:54.221398 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.212484    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 31 12:41:54.246089 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.243439    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:41:54.246089 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.244248    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:41:54.387379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.384738    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:41:54.387379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.385497    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:41:54.701577 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.701469    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:54.722137 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.722093    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:54.722797 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.722764    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:41:54.739927 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.722967    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:41:54.767767 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.722991    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:41:54.768258 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.723067    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:41:54.768483 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.768465    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:41:54.768623 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.768612    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:41:54.768726 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.768717    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:41:54.772374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.723525    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:54.785377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.773786    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:41:54.827705 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.827665    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:41:54.828260 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.828231    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:41:54.828653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.828631    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:41:54.829184 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.829164    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:41:54.829365 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.829335    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:55.121856 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.121821    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:41:55.122831 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.847833    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:55.124030 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:54.942490    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:55.146850 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.124599    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:41:55.147680 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.034589    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:55.148943 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.148922    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:41:55.187392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.172540    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:55.187674 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.172573    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:55.187842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.172609    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:55.324529 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.324481    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:41:55.387827 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.387739    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:55.778634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.778502    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:55.967712 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.967672    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:55.972407 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.972253    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:55.973680 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.973651    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:55.974855 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:55.974831    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.003058 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.003019    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.006671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.006635    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:41:56.009523 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.009473    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.186521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.186466    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.247758 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.190815    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:41:56.248118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.190849    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:41:56.273379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.248415    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:41:56.345403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.328887    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.492458 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.473977    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.513436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.502277    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.513918 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.438996    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.790437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.762736    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 31 12:41:56.820526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.815472    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:41:56.858488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.858443    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:56.948382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.944932    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:41:56.975023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:56.974969    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 31 12:41:57.077777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.077740    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:57.104590 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.104551    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:41:57.370774 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.108523    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:41:57.372049 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.108582    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:41:57.373015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.108616    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:41:57.381449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.112425    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:41:57.381905 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.112472    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:41:57.382277 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.216597    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:57.382477 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.271462    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:57.405374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.349346    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:41:57.590652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.590604    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:41:57.593494 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.593468    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:41:57.843380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:57.835813    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:41:58.149482 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.120588    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:58.288030 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.286780    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:58.381411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.378052    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:58.402183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.383198    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:58.402183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.384648    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:41:58.402183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.384983    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:41:58.402183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.391427    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:58.402183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.391815    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:41:58.468778 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.468744    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:41:58.473035 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.473012    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:41:58.474043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.469640    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:41:58.502390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.489606    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:41:58.654472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.652631    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:41:58.746891 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.746845    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:58.818541 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.818510    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:41:58.830000 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.829970    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:41:58.839731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.839692    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:41:58.840589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.840563    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:41:58.858388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.852047    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:41:58.904123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.885716    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:41:58.904575 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.904496    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:41:58.904791 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.888667    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:41:58.988536 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.988219    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:41:58.989220 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:41:58.989196    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:42:07.978037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:07.977996    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"]
Oct 31 12:42:07.983287 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:07.983259    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.006036 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.005999    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"]
Oct 31 12:42:08.114367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.111391    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7888a0-116d-4036-98f3-c28c6be8e61e-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.114367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.111491    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.114367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.111550    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.114367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.111646    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.114367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.111835    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.114367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.111863    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p5sd\" (UniqueName: \"kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-kube-api-access-8p5sd\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.114367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.111891    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.114367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.112008    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.114367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.112056    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.213391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.213336    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.213925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.213403    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p5sd\" (UniqueName: \"kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-kube-api-access-8p5sd\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.213925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.213438    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.213925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.213525    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.213925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.213567    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.213925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.213662    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7888a0-116d-4036-98f3-c28c6be8e61e-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.213925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.213703    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.213925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.213746    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.213925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.213803    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.213925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.213905    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.214519 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.213942    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.214519 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.213996    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.214519 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.214199    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.214519 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.214502    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7888a0-116d-4036-98f3-c28c6be8e61e-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.216476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.216451    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.216605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.216584    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.225770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.225743    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p5sd\" (UniqueName: \"kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-kube-api-access-8p5sd\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.225964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.225946    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.301404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.301297    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:08.606487 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.606454    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"]
Oct 31 12:42:08.607233 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:42:08.607207    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7888a0_116d_4036_98f3_c28c6be8e61e.slice/crio-56e74be6a309cc337be7746848c26e5bd2c0f758d19216db7f9403646cd995c1 WatchSource:0}: Error finding container 56e74be6a309cc337be7746848c26e5bd2c0f758d19216db7f9403646cd995c1: Status 404 returned error can't find the container with id 56e74be6a309cc337be7746848c26e5bd2c0f758d19216db7f9403646cd995c1
Oct 31 12:42:08.609776 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.609744    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:08.609854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.609808    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:08.609854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:08.609839    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:09.598744 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:09.598705    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh" event={"ID":"fb7888a0-116d-4036-98f3-c28c6be8e61e","Type":"ContainerStarted","Data":"c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2"}
Oct 31 12:42:09.599167 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:09.598770    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh" event={"ID":"fb7888a0-116d-4036-98f3-c28c6be8e61e","Type":"ContainerStarted","Data":"56e74be6a309cc337be7746848c26e5bd2c0f758d19216db7f9403646cd995c1"}
Oct 31 12:42:09.641271 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:09.641222    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh" podStartSLOduration=2.641208496 podStartE2EDuration="2.641208496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:09.640710899 +0000 UTC m=+854.381137116" watchObservedRunningTime="2026-10-31 12:42:09.641208496 +0000 UTC m=+854.381634711"
Oct 31 12:42:10.302426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:10.302393    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:10.303937 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:10.303907    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 12:42:10.304084 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:10.303972    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh" podUID="fb7888a0-116d-4036-98f3-c28c6be8e61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 12:42:11.302418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:11.302383    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 12:42:11.302817 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:11.302446    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh" podUID="fb7888a0-116d-4036-98f3-c28c6be8e61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 12:42:12.301917 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:12.301879    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 12:42:12.302095 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:12.301936    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh" podUID="fb7888a0-116d-4036-98f3-c28c6be8e61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 12:42:13.307379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:13.307135    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 12:42:13.307379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:13.307208    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh" podUID="fb7888a0-116d-4036-98f3-c28c6be8e61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 12:42:14.304375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:14.303784    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 12:42:14.304375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:14.303848    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh" podUID="fb7888a0-116d-4036-98f3-c28c6be8e61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 12:42:15.302688 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:15.302653    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 12:42:15.303147 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:15.302712    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh" podUID="fb7888a0-116d-4036-98f3-c28c6be8e61e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 12:42:16.306195 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:16.306171    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:16.306568 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:16.306468    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:16.307164 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:16.307147    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 12:42:22.057452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.057408    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"]
Oct 31 12:42:22.090705 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.090676    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.092330 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.092301    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"]
Oct 31 12:42:22.223117 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.223073    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:22.284382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.271472    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.291382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.285666    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btv2j\" (UniqueName: \"kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-kube-api-access-btv2j\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.291657 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.291631    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a849ea26-cba4-4899-87a0-9eb688e0cfae-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.291786 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.291772    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.291902 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.291888    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.292042 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.292028    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.300556 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.300518    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.300787 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.300769    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.305499 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.305465    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.329588 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.329489    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:42:22.375386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.374447    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:22.454446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.401430    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"]
Oct 31 12:42:22.454446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.414460    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:42:22.454446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.415970    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btv2j\" (UniqueName: \"kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-kube-api-access-btv2j\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.454446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.416034    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a849ea26-cba4-4899-87a0-9eb688e0cfae-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.454446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.416082    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.454446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.416120    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.454446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.416167    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.454446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.416260    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.454446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.416288    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.454446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.416367    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.454446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.416515    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.454446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.417042    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-certs\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.464490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.456013    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a849ea26-cba4-4899-87a0-9eb688e0cfae-istiod-ca-cert\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.464490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.456310    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.464490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.456600    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-credential-socket\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.464490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.457143    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-data\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.496444 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.496181    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.548380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.523705    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:22.553695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.533972    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"]
Oct 31 12:42:22.630836 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.630697    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.630836 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.630818    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.631067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.630896    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.631067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.631053    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e183677e-396a-41f6-aa42-1184c82d520b-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.631186 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.631086    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.631186 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.631122    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db6r5\" (UniqueName: \"kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-kube-api-access-db6r5\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.631303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.631222    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.631303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.631283    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e183677e-396a-41f6-aa42-1184c82d520b-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.631436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.631311    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.724084 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.694410    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"]
Oct 31 12:42:22.805380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.750511    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.947428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.936284    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.947428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.936461    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e183677e-396a-41f6-aa42-1184c82d520b-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.947428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.936516    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.947428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.936569    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db6r5\" (UniqueName: \"kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-kube-api-access-db6r5\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.947428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.936677    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.947428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.936736    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e183677e-396a-41f6-aa42-1184c82d520b-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.947428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.936769    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.947428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.936933    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.947428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.937680    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.947428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.755050    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-podinfo\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.947428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.946980    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.947428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.769033    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:22.978373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.951699    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:22.978373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.956096    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btv2j\" (UniqueName: \"kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-kube-api-access-btv2j\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:22.978373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.969134    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-envoy\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:23.004380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.999467    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:23.034387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.033375    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:23.040387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.039568    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-token\") pod \"e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") " pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:23.040387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.040051    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:42:23.040627 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.040316    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:23.040627 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.934479    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:23.049379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.934646    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:23.057422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.057391    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e183677e-396a-41f6-aa42-1184c82d520b-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:23.081816 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.081785    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"]
Oct 31 12:42:23.082694 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.934551    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:42:23.098989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.098953    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:23.111466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:22.934609    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:42:23.236229 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.236123    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.236529 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.236492    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.237906 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.237882    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.238139 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.238103    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.249715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.249689    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:23.251396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.251344    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.251671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.251650    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.251883 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.251866    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.262416 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.251998    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:23.284796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.263851    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdlbv\" (UniqueName: \"kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-kube-api-access-pdlbv\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.284796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.263951    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.284796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.263979    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:42:23.284796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.276575    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:23.284796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.277310    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:42:23.285202 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.285040    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e183677e-396a-41f6-aa42-1184c82d520b-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:23.299555 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:42:23.287265    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:42:23.368777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.344097    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:42:23.368777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.353315    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:42:23.368777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.366624    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.368777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.366692    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.368777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.366782    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.368777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.366848    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.368777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.366903    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.368777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.366943    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdlbv\" (UniqueName: \"kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-kube-api-access-pdlbv\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.368777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.366986    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.368777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.367063    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.368777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.367099    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.515527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.398948    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-data\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.515527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.399031    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:23.515527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.399306    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:42:23.515527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.403178    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db6r5\" (UniqueName: \"kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-kube-api-access-db6r5\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:23.515527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.505133    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-token\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.603422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.545852    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:42:23.603422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.573444    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:23.603422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.593852    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.603422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.594143    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-certs\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.603422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.594399    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-credential-socket\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.603422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.595269    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istiod-ca-cert\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.604045 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.603976    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdlbv\" (UniqueName: \"kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-kube-api-access-pdlbv\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.648709 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.648428    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:23.697378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.679442    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-podinfo\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.784434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.759217    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:42:23.784760 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.784731    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:42:23.874379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.850896    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-envoy\") pod \"e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") " pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:23.946577 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:23.939466    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Oct 31 12:42:24.021709 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.021666    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:24.053426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.053205    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:24.055170 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.055144    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:42:24.094623 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.090435    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:24.100814 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.100773    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:42:24.164403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.113887    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:24.164403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.156034    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:24.180385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.172817    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:42:24.180385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.172891    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:42:24.218490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.187931    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:24.218490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.193273    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:42:24.308436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.301465    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 12:42:24.308436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.301882    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:24.308436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.303574    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:42:24.312700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.312657    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:42:24.324531 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.324482    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:42:24.340481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.340437    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Oct 31 12:42:24.340716 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.340437    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:24.364448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.359600    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:42:24.364448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.360655    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:24.419142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.415524    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:24.563847 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.552584    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:24.595576 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.595519    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:42:24.759208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:24.758122    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:25.130380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.101640    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:25.130380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.104218    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:42:25.130380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.114790    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:25.189585 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.189548    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:42:25.190157 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.190127    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:42:25.213963 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.213709    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded"
Oct 31 12:42:25.213963 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.213802    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Oct 31 12:42:25.258147 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.250041    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:42:25.262477 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.258447    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:25.304406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.303388    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:25.304406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.303680    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 12:42:25.321396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.320407    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:25.408586 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.403578    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:25.408586 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.403970    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:42:25.435376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.433674    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:42:25.466223 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.466184    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:25.489454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.489409    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:25.489747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.489720    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:25.646646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.608570    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:42:25.646646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:25.614626    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:26.084982 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.084939    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Oct 31 12:42:26.160415 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.160371    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:42:26.161439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.161410    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:42:26.187751 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.166851    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" event={"ID":"e183677e-396a-41f6-aa42-1184c82d520b","Type":"ContainerStarted","Data":"658fbefd5e666645e4f803041f2ce918ea7f52c691b1b8c1469aa1da5a0906a3"}
Oct 31 12:42:26.187751 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.166907    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"]
Oct 31 12:42:26.214310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.214261    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:42:26.240380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.226409    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:42:26.275981 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.266431    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:26.354865 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.354824    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:26.376378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.367468    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:42:26.407325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.407277    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:26.567406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.407578    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:26.574810 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.574752    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:26.843463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.575615    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:26.843785 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:42:26.575910    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47c3e79_0ee1_4b08_ae9e_87e3507703f5.slice/crio-a12625931439c1fc310fd524df3bfbfef1e1c56c657d773cb139eda248d69770 WatchSource:0}: Error finding container a12625931439c1fc310fd524df3bfbfef1e1c56c657d773cb139eda248d69770: Status 404 returned error can't find the container with id a12625931439c1fc310fd524df3bfbfef1e1c56c657d773cb139eda248d69770
Oct 31 12:42:26.843974 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.717505    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:26.844121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.759886    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:42:26.844295 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.761239    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:42:26.844485 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.772864    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:26.844671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.772971    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 31 12:42:26.844781 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.793308    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:26.848376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.806030    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:26.882820 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.882766    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:26.907325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.884408    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:26.909229 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.909182    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:26.996116 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:26.996066    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:27.214290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.179226    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.214290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.179260    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.214290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.179290    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.214290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.192453    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.214290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.201152    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"]
Oct 31 12:42:27.214290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.213078    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"]
Oct 31 12:42:27.227594 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.226582    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.253377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.250431    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.267870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.267826    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.273917 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.273887    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:42:27.334402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.331096    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" event={"ID":"e183677e-396a-41f6-aa42-1184c82d520b","Type":"ContainerStarted","Data":"3950c3ffd9b4906c4612d07c7b7758690ae38a84edf1b979cef879cc2bba0ef2"}
Oct 31 12:42:27.379414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.379375    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 12:42:27.380849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.380817    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:42:27.576385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.565671    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podStartSLOduration=5.565647721 podStartE2EDuration="5.565647721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:27.399300015 +0000 UTC m=+872.139726417" watchObservedRunningTime="2026-10-31 12:42:27.565647721 +0000 UTC m=+872.306073938"
Oct 31 12:42:27.576385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.568185    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:42:27.576385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.568570    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:42:27.706379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.672022    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 12:42:27.706672 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.706636    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 12:42:27.706846 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.705691    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:27.710375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.707406    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" event={"ID":"c47c3e79-0ee1-4b08-ae9e-87e3507703f5","Type":"ContainerStarted","Data":"a12625931439c1fc310fd524df3bfbfef1e1c56c657d773cb139eda248d69770"}
Oct 31 12:42:27.796293 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.796244    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" event={"ID":"a849ea26-cba4-4899-87a0-9eb688e0cfae","Type":"ContainerStarted","Data":"e5c0677656f437fa1cfebff27753805dd985e9ab76bacac3f0dc55bb9847bef4"}
Oct 31 12:42:27.852391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.851916    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:27.857428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.854141    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:42:27.894307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.858470    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:42:27.896476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:27.858505    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:42:28.217388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.215467    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:28.217388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.215812    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:42:28.236248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.229941    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:28.270164 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.270121    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:28.275070 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.274378    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:42:28.413704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.410744    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 12:42:28.413704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.413067    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded"
Oct 31 12:42:28.556292 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.556238    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:28.575059 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.575011    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:28.577082 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.575612    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:42:28.613681 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.611094    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:42:28.654740 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.654705    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 12:42:28.655034 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.655000    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 12:42:28.901462 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.870946    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:42:28.930422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.905217    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:42:28.936829 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.936790    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:42:28.939194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.937129    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:42:29.007688 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:28.999638    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:42:29.032384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.031366    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:42:29.123258 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.123229    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:42:29.127256 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.125912    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:42:29.137086 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.137059    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:42:29.223437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.210962    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:42:29.230932 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.225036    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:29.270701 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.270664    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" event={"ID":"c47c3e79-0ee1-4b08-ae9e-87e3507703f5","Type":"ContainerStarted","Data":"43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a"}
Oct 31 12:42:29.320423 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.320388    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" event={"ID":"a849ea26-cba4-4899-87a0-9eb688e0cfae","Type":"ContainerStarted","Data":"32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a"}
Oct 31 12:42:29.320906 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.320882    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:42:29.323116 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.323095    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:42:29.323280 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.323268    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:42:29.493680 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.493588    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:42:29.529462 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.503742    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:42:29.636918 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.636884    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:42:29.655025 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.654981    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 12:42:29.655183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.655044    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 12:42:29.655415 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.655327    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" podStartSLOduration=7.655311073 podStartE2EDuration="7.655311073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:29.634497655 +0000 UTC m=+874.374923917" watchObservedRunningTime="2026-10-31 12:42:29.655311073 +0000 UTC m=+874.395737290"
Oct 31 12:42:29.898578 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:29.862263    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" podStartSLOduration=7.862242752 podStartE2EDuration="7.862242752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:29.790980925 +0000 UTC m=+874.531407142" watchObservedRunningTime="2026-10-31 12:42:29.862242752 +0000 UTC m=+874.602668969"
Oct 31 12:42:30.160481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:30.139838    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:30.160759 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:30.144520    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 12:42:30.160948 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:30.160917    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 12:42:30.270533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:30.270497    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:30.271208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:30.271174    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 12:42:30.271398 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:30.271372    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 12:42:30.653862 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:30.653831    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 12:42:30.654069 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:30.653886    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 12:42:31.085876 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:31.085836    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 12:42:31.086164 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:31.086137    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 12:42:31.276659 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:31.276622    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 12:42:31.277272 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:31.277240    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 12:42:31.683677 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:31.676989    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 12:42:31.683677 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:31.677052    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 12:42:32.091377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:32.090470    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 12:42:32.091377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:32.090532    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 12:42:32.278249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:32.278213    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 12:42:32.293963 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:32.293913    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 12:42:32.655267 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:32.655229    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 12:42:32.655487 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:32.655292    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 12:42:33.086184 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:33.086148    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 12:42:33.102482 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:33.102437    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 12:42:33.270778 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:33.270739    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 12:42:33.271054 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:33.271028    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 12:42:33.658770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:33.658730    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 12:42:33.659380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:33.659325    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 12:42:34.103377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:34.102759    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 12:42:34.103377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:34.102835    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 12:42:34.274376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:34.268294    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 12:42:34.274376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:34.268386    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 12:42:34.654333 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:34.654284    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 12:42:34.654533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:34.654379    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 12:42:35.088835 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:35.088799    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 12:42:35.089461 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:35.089424    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 12:42:35.267380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:35.265920    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 12:42:35.267380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:35.265988    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 12:42:35.653651 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:35.653598    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 12:42:35.653855 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:35.653666    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 12:42:36.085619 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:36.085571    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 12:42:36.085830 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:36.085645    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 12:42:36.265436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:36.265394    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 12:42:36.265828 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:36.265475    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 12:42:36.654678 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:36.654633    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 12:42:36.654898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:36.654717    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 12:42:37.354846 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:37.354806    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:37.356762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:37.356730    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:37.358443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:37.358421    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 12:42:37.713714 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:37.713630    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:37.717481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:37.716784    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:37.720323 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:37.720296    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:42:37.834703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:37.834665    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:37.835728 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:37.835701    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:37.841033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:37.841008    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:42:37.883689 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:37.883660    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"]
Oct 31 12:42:37.889251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:37.889221    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:37.937420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:37.933403    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"]
Oct 31 12:42:38.088170 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.088132    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.088330 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.088179    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.088330 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.088261    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.088504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.088436    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbfr2\" (UniqueName: \"kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-kube-api-access-cbfr2\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.088504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.088491    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.088587 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.088516    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.088660 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.088644    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.088703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.088671    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.088703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.088689    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.190575 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.190535    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.190575 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.190579    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.190782 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.190602    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.190782 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.190713    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.190782 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.190750    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.190933 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.190792    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.190933 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.190898    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbfr2\" (UniqueName: \"kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-kube-api-access-cbfr2\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.191092 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.190958    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.191092 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.191002    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.191092 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.191009    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-certs\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.191092 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.191059    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-data\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.191414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.191345    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.191488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.191414    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-credential-socket\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.191521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.191507    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istiod-ca-cert\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.193032 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.193014    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-envoy\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.193496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.193476    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-podinfo\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.203012 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.202982    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-token\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.203427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.203405    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbfr2\" (UniqueName: \"kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-kube-api-access-cbfr2\") pod \"e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") " pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.255322 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.255282    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:38.628375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.617196    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"]
Oct 31 12:42:38.665364 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.665314    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:38.665504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.665430    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:38.665504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.665467    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:42:38.684878 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:38.684844    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" event={"ID":"2054e1e9-a58c-4fe9-806c-64959e42a5d4","Type":"ContainerStarted","Data":"5cb1cdf861b0be26dd28471b635f2e106536777588598bc1c1dff477cd836c27"}
Oct 31 12:42:39.694222 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:39.694181    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" event={"ID":"2054e1e9-a58c-4fe9-806c-64959e42a5d4","Type":"ContainerStarted","Data":"0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394"}
Oct 31 12:42:39.730824 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:39.730775    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" podStartSLOduration=2.7307620569999997 podStartE2EDuration="2.730762057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:39.726464787 +0000 UTC m=+884.466891014" watchObservedRunningTime="2026-10-31 12:42:39.730762057 +0000 UTC m=+884.471188273"
Oct 31 12:42:40.256045 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:40.256013    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:40.282534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:40.282495    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 12:42:40.282747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:40.282574    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" podUID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 12:42:41.255857 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:41.255825    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 12:42:41.256246 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:41.255874    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" podUID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 12:42:42.255949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:42.255918    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 12:42:42.256397 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:42.255973    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" podUID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 12:42:43.255839 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:43.255804    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 12:42:43.255984 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:43.255882    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" podUID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 12:42:44.255965 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:44.255927    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 12:42:44.256147 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:44.255986    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" podUID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 12:42:45.259764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:45.259727    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 12:42:45.280173 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:45.280133    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" podUID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 12:42:46.556598 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:46.556564    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:46.557064 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:46.556956    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:46.558288 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:46.558268    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:42:49.342922 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:49.340458    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:49.343521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:49.343427    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 12:42:49.406509 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:49.406448    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:49.576023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:49.575972    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:49.582458 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:49.582415    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:42:49.668596 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:49.668485    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:49.914467 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:49.898079    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:50.249710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:50.249660    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:50.341537 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:50.341465    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:50.367379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:50.342679    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:42:50.384743 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:50.384699    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:50.385814 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:50.385785    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:50.386323 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:50.386304    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:42:50.442379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:50.423245    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:42:50.448495 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:50.448458    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:42:50.935379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:50.934624    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:50.958382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:50.936469    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:42:50.965486 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:50.965443    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:50.972393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:50.971578    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:42:51.021771 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.021727    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:42:51.040002 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.039974    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:42:51.047603 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.047558    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:42:51.072339 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.072309    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:42:51.139008 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.116502    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:51.153440 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.148448    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:51.184140 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.184088    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:51.290430 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.260452    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:51.290430 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.260915    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:42:51.347489 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.347445    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:42:51.371380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.366507    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:51.386281 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.385238    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:51.414572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.390528    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:42:51.414572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.397861    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:51.532505 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.532455    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:51.936807 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.936759    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:51.972690 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:51.972638    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:42:52.062502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.062458    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:52.086876 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.082476    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:52.143379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.139436    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:42:52.143379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.140719    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:42:52.160527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.160482    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:52.170640 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.170574    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:42:52.196457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.185443    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:52.196457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.185784    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:42:52.198444 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.198415    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:42:52.325418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.295460    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:42:52.348646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.348610    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:42:52.375381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.374828    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:42:52.376585 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.376163    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:42:52.412408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.407530    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:52.468459 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.459017    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:42:52.532990 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.532933    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:52.939403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.939329    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:42:52.991309 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:52.991270    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.084498 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.084451    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.141108 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.140538    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.167703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.166458    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.197783 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.197680    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.247602 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.247550    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.322405 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.321833    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:42:53.374387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.350434    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.378386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.375228    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:42:53.425626 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.425582    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.456536 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:42:53.455548    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:42:53.537222 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.537169    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.879378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.848695    5663 patch_prober.go:29] interesting pod/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:42:53.879378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.848777    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.879378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.849437    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:53.955431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:53.941462    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.002630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.002561    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.105418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.105367    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.179062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.178940    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.191674 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.182035    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:42:54.214489 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.191487    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.215179 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.209650    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.344370 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.344290    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:42:54.352822 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.352777    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.377503 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.377449    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:42:54.432588 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.432478    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.538810 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.538232    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.679723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.679477    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:54.830761 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.817155    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:42:54.951552 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.951510    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:42:54.962559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:54.961485    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:42:55.008413 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.008367    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:42:55.074199 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.074155    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:55.114869 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.114773    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:55.157341 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.157295    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:55.191470 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.191430    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:55.212268 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.212226    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:55.216745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.216707    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:55.217261 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.216881    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:42:55.275524 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.253223    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"]
Oct 31 12:42:55.481419 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.471234    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:55.524381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.500768    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:42:55.901610 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.599200    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:55.960620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.599239    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:42:55.960620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.951197    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"
Oct 31 12:42:56.000944 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.961899    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.000944 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:55.964161    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-59fd8a28\""
Oct 31 12:42:56.032462 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.017463    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.137069 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.023700    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-59fd8a28-dockercfg-jsp6g\""
Oct 31 12:42:56.137561 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.102805    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7j8g\" (UniqueName: \"kubernetes.io/projected/8403b6b0-9749-4b66-a651-f1db7c6b6e71-kube-api-access-z7j8g\") pod \"payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm\" (UID: \"8403b6b0-9749-4b66-a651-f1db7c6b6e71\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"
Oct 31 12:42:56.138037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.137999    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8403b6b0-9749-4b66-a651-f1db7c6b6e71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm\" (UID: \"8403b6b0-9749-4b66-a651-f1db7c6b6e71\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"
Oct 31 12:42:56.183573 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.102882    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:42:56.183573 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.164324    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:42:56.183573 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.136525    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.223220 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.223168    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.223816 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.223693    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.223816 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.223707    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.603380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.590823    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:42:56.603380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.595083    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.620822 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.620767    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.627984 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.627899    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7j8g\" (UniqueName: \"kubernetes.io/projected/8403b6b0-9749-4b66-a651-f1db7c6b6e71-kube-api-access-z7j8g\") pod \"payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm\" (UID: \"8403b6b0-9749-4b66-a651-f1db7c6b6e71\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"
Oct 31 12:42:56.630182 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.630152    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8403b6b0-9749-4b66-a651-f1db7c6b6e71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm\" (UID: \"8403b6b0-9749-4b66-a651-f1db7c6b6e71\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"
Oct 31 12:42:56.636793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.636757    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.656130 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.656088    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.787411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.780986    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:42:56.908230 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.908131    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:56.949914 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:56.938574    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-59fd8a28\""
Oct 31 12:42:57.033942 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.033882    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:57.049600 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.043512    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:57.127385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.127329    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8403b6b0-9749-4b66-a651-f1db7c6b6e71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm\" (UID: \"8403b6b0-9749-4b66-a651-f1db7c6b6e71\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"
Oct 31 12:42:57.161429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.147445    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:57.265389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.262471    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:57.270157 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.269851    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:42:57.287414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.285469    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:57.287414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.285878    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:57.301193 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.301115    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:42:57.302331 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.302297    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:57.349765 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:42:57.349729    5663 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.437s"
Oct 31 12:42:57.350061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.350043    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:42:57.350207 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.350192    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"]
Oct 31 12:42:57.350505 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.350487    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:42:57.350689 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.350676    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:42:57.350841 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.350829    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:42:57.441078 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.440982    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:57.550603 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.550566    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7j8g\" (UniqueName: \"kubernetes.io/projected/8403b6b0-9749-4b66-a651-f1db7c6b6e71-kube-api-access-z7j8g\") pod \"payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm\" (UID: \"8403b6b0-9749-4b66-a651-f1db7c6b6e71\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"
Oct 31 12:42:57.707616 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.707518    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:57.816386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.782340    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:42:57.865253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.865224    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:42:57.872205 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.872183    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:42:57.897378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.893941    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:42:57.902406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.899602    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:42:57.990369 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.881323    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-59fd8a28-dockercfg-jsp6g\""
Oct 31 12:42:58.020677 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.927480    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:42:58.039378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.950495    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:42:58.039722 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:57.893427    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"
Oct 31 12:42:58.132374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:58.128839    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:42:58.132374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:58.131615    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:42:58.135730 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:58.135704    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:42:58.158924 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:58.158845    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:42:58.516049 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:58.516021    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"]
Oct 31 12:42:58.517114 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:42:58.517084    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8403b6b0_9749_4b66_a651_f1db7c6b6e71.slice/crio-a7398309c36f8ab26dc26b90a5a7bac21da40f90801d1fa1256a63b5f6066447 WatchSource:0}: Error finding container a7398309c36f8ab26dc26b90a5a7bac21da40f90801d1fa1256a63b5f6066447: Status 404 returned error can't find the container with id a7398309c36f8ab26dc26b90a5a7bac21da40f90801d1fa1256a63b5f6066447
Oct 31 12:42:59.487801 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:59.487767    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm" event={"ID":"8403b6b0-9749-4b66-a651-f1db7c6b6e71","Type":"ContainerStarted","Data":"324ff06430cbdc421616c07ecfef1d1d4f525b8f20a74ca801014630920dbc48"}
Oct 31 12:42:59.488232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:59.487816    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm" event={"ID":"8403b6b0-9749-4b66-a651-f1db7c6b6e71","Type":"ContainerStarted","Data":"a7398309c36f8ab26dc26b90a5a7bac21da40f90801d1fa1256a63b5f6066447"}
Oct 31 12:42:59.488232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:59.487910    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"
Oct 31 12:42:59.508173 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:42:59.508124    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm" podStartSLOduration=4.508110864 podStartE2EDuration="4.508110864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:42:59.504853504 +0000 UTC m=+904.245279719" watchObservedRunningTime="2026-10-31 12:42:59.508110864 +0000 UTC m=+904.248537104"
Oct 31 12:43:02.838797 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:02.838704    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"]
Oct 31 12:43:02.845187 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:02.845159    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:02.860574 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:02.856346    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-59fd8a28-serving-cert\""
Oct 31 12:43:02.891737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:02.890550    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"]
Oct 31 12:43:02.970278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:02.965510    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-openshift-service-ca\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:02.970278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:02.965622    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-maas-api-tls\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:02.970278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:02.965690    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-metrics-certs\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:02.970278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:02.965731    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcczb\" (UniqueName: \"kubernetes.io/projected/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-kube-api-access-fcczb\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:03.076543 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:03.067405    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-metrics-certs\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:03.077456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:03.077416    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcczb\" (UniqueName: \"kubernetes.io/projected/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-kube-api-access-fcczb\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:03.077979 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:03.077955    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-openshift-service-ca\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:03.089422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:03.080881    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-openshift-service-ca\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:03.089877 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:03.089808    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-maas-api-tls\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:03.092040 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:43:03.090294    5663 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-serving-cert: secret "maas-api-e2e-ait-59fd8a28-serving-cert" not found
Oct 31 12:43:03.092040 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:43:03.090436    5663 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-maas-api-tls podName:4bb42103-e803-4ed2-a606-2fb5c3dc3fef nodeName:}" failed. No retries permitted until 2026-10-31 12:43:03.590407525 +0000 UTC m=+908.330833721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-maas-api-tls") pod "maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf" (UID: "4bb42103-e803-4ed2-a606-2fb5c3dc3fef") : secret "maas-api-e2e-ait-59fd8a28-serving-cert" not found
Oct 31 12:43:03.092825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:03.092800    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-metrics-certs\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:03.092965 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:03.092931    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcczb\" (UniqueName: \"kubernetes.io/projected/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-kube-api-access-fcczb\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:03.650381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:03.649689    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-maas-api-tls\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:03.656376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:03.654192    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-maas-api-tls\") pod \"maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:03.793627 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:03.793594    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:04.104438 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:43:04.104398    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb42103_e803_4ed2_a606_2fb5c3dc3fef.slice/crio-d0b0dbe5396e6613ea936a490f2ed9d86f74accb4f97c55bd0587906ea90c537 WatchSource:0}: Error finding container d0b0dbe5396e6613ea936a490f2ed9d86f74accb4f97c55bd0587906ea90c537: Status 404 returned error can't find the container with id d0b0dbe5396e6613ea936a490f2ed9d86f74accb4f97c55bd0587906ea90c537
Oct 31 12:43:04.105240 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:04.105212    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"]
Oct 31 12:43:04.106366 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:04.106334    5663 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:43:04.556758 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:04.556726    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf" event={"ID":"4bb42103-e803-4ed2-a606-2fb5c3dc3fef","Type":"ContainerStarted","Data":"d0b0dbe5396e6613ea936a490f2ed9d86f74accb4f97c55bd0587906ea90c537"}
Oct 31 12:43:05.566308 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:05.566266    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf" event={"ID":"4bb42103-e803-4ed2-a606-2fb5c3dc3fef","Type":"ContainerStarted","Data":"6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195"}
Oct 31 12:43:05.566829 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:05.566342    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:05.587890 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:05.587835    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf" podStartSLOduration=3.040821439 podStartE2EDuration="3.587818157s" totalImagesPullingTime="546.996718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:05.585833802 +0000 UTC m=+910.326260019" watchObservedRunningTime="2026-10-31 12:43:05.587818157 +0000 UTC m=+910.328244387"
Oct 31 12:43:06.950221 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:06.950191    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"]
Oct 31 12:43:06.954619 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:06.954597    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:06.959559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:06.959536    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d5e8c3f2-serving-cert\""
Oct 31 12:43:06.966290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:06.966264    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"]
Oct 31 12:43:07.057167 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.057135    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-maas-api-tls\") pod \"maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.057377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.057264    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f28622af-2097-40b7-bf72-01b39d2bdfd7-openshift-service-ca\") pod \"maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.057377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.057323    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v2tx\" (UniqueName: \"kubernetes.io/projected/f28622af-2097-40b7-bf72-01b39d2bdfd7-kube-api-access-8v2tx\") pod \"maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.057515 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.057436    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-metrics-certs\") pod \"maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.159445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.159412    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-maas-api-tls\") pod \"maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.159626 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.159499    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f28622af-2097-40b7-bf72-01b39d2bdfd7-openshift-service-ca\") pod \"maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.159626 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.159526    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v2tx\" (UniqueName: \"kubernetes.io/projected/f28622af-2097-40b7-bf72-01b39d2bdfd7-kube-api-access-8v2tx\") pod \"maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.159626 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.159553    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-metrics-certs\") pod \"maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.160385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.160321    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f28622af-2097-40b7-bf72-01b39d2bdfd7-openshift-service-ca\") pod \"maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.162043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.162025    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-metrics-certs\") pod \"maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.162266 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.162242    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-maas-api-tls\") pod \"maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.202447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.192252    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v2tx\" (UniqueName: \"kubernetes.io/projected/f28622af-2097-40b7-bf72-01b39d2bdfd7-kube-api-access-8v2tx\") pod \"maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.271746 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.271708    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:07.886632 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:07.886590    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"]
Oct 31 12:43:08.222232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.222161    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"]
Oct 31 12:43:08.229124 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.229100    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.252150 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.252113    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"]
Oct 31 12:43:08.358570 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.358548    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.358652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.358592    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.358712 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.358701    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.358865 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.358837    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b55a494-6e14-4759-b8d7-53bf788aab78-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.358968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.358949    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.359025 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.358985    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.359025 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.359017    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9nj8\" (UniqueName: \"kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-kube-api-access-n9nj8\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.359296 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.359281    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.359342 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.359309    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.460650 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.460607    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.460650 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.460654    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.460888 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.460715    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.460888 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.460818    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b55a494-6e14-4759-b8d7-53bf788aab78-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.461002 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.460910    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.461002 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.460950    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.461109 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.460997    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nj8\" (UniqueName: \"kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-kube-api-access-n9nj8\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.461109 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.461056    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.461109 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.461059    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-credential-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.461109 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.461087    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.461303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.461169    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-data\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.461404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.461380    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-certs\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.461596 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.461578    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-socket\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.461668 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.461601    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b55a494-6e14-4759-b8d7-53bf788aab78-istiod-ca-cert\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.463329 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.463294    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-envoy\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.463966 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.463947    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-podinfo\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.473147 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.473091    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-token\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.473265 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.473248    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nj8\" (UniqueName: \"kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-kube-api-access-n9nj8\") pod \"e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") " pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.557939 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.557912    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:08.592173 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.592133    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq" event={"ID":"f28622af-2097-40b7-bf72-01b39d2bdfd7","Type":"ContainerStarted","Data":"1d705e98d60caefcb7e518b74fe1b6e4b544386edd876ce1a432f13ed4386a90"}
Oct 31 12:43:08.592392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.592198    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq" event={"ID":"f28622af-2097-40b7-bf72-01b39d2bdfd7","Type":"ContainerStarted","Data":"157aee3886c99e509d2fdde5e40f852c376bb18e34d2832eb99a76c23bdc513d"}
Oct 31 12:43:08.592392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.592242    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:08.623056 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:08.620150    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq" podStartSLOduration=2.154354341 podStartE2EDuration="2.620130612s" totalImagesPullingTime="465.776271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:08.618637743 +0000 UTC m=+913.359063960" watchObservedRunningTime="2026-10-31 12:43:08.620130612 +0000 UTC m=+913.360556828"
Oct 31 12:43:09.004026 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:43:09.003982    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b55a494_6e14_4759_b8d7_53bf788aab78.slice/crio-b3f9a6a4c40b8dc08e708f9ad3b06a1f66d19342d02bfd74fd46a4c952cccb87 WatchSource:0}: Error finding container b3f9a6a4c40b8dc08e708f9ad3b06a1f66d19342d02bfd74fd46a4c952cccb87: Status 404 returned error can't find the container with id b3f9a6a4c40b8dc08e708f9ad3b06a1f66d19342d02bfd74fd46a4c952cccb87
Oct 31 12:43:09.017767 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.010171    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:09.017767 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.010264    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:09.017767 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.010305    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:09.026582 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.026549    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"]
Oct 31 12:43:09.152747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.152703    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"]
Oct 31 12:43:09.158144 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.158119    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.192195 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.190189    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b849a4c4-serving-cert\""
Oct 31 12:43:09.192637 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.192609    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"]
Oct 31 12:43:09.305597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.305335    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-maas-api-tls\") pod \"maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.306000 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.305672    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqrjz\" (UniqueName: \"kubernetes.io/projected/87974688-1b9a-4778-8d77-158f5cdfa4aa-kube-api-access-qqrjz\") pod \"maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.306000 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.305779    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-metrics-certs\") pod \"maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.306000 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.305969    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87974688-1b9a-4778-8d77-158f5cdfa4aa-openshift-service-ca\") pod \"maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.421380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.414406    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqrjz\" (UniqueName: \"kubernetes.io/projected/87974688-1b9a-4778-8d77-158f5cdfa4aa-kube-api-access-qqrjz\") pod \"maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.421380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.414541    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-metrics-certs\") pod \"maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.421380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.414688    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87974688-1b9a-4778-8d77-158f5cdfa4aa-openshift-service-ca\") pod \"maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.421380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.414793    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-maas-api-tls\") pod \"maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.486377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.461125    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87974688-1b9a-4778-8d77-158f5cdfa4aa-openshift-service-ca\") pod \"maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.486639 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.467114    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-metrics-certs\") pod \"maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.654438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.602590    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-maas-api-tls\") pod \"maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.654438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.617113    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqrjz\" (UniqueName: \"kubernetes.io/projected/87974688-1b9a-4778-8d77-158f5cdfa4aa-kube-api-access-qqrjz\") pod \"maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.796315 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.796278    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" event={"ID":"1b55a494-6e14-4759-b8d7-53bf788aab78","Type":"ContainerStarted","Data":"68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37"}
Oct 31 12:43:09.796724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.796638    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" event={"ID":"1b55a494-6e14-4759-b8d7-53bf788aab78","Type":"ContainerStarted","Data":"b3f9a6a4c40b8dc08e708f9ad3b06a1f66d19342d02bfd74fd46a4c952cccb87"}
Oct 31 12:43:09.841429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.833433    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:09.845943 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:09.845809    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" podStartSLOduration=1.845788018 podStartE2EDuration="1.845788018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:09.841259594 +0000 UTC m=+914.581685810" watchObservedRunningTime="2026-10-31 12:43:09.845788018 +0000 UTC m=+914.586214235"
Oct 31 12:43:10.348954 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:10.348738    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"]
Oct 31 12:43:10.568935 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:10.568903    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:10.586030 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:10.583501    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 12:43:10.586030 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:10.583604    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 12:43:10.651345 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:10.651265    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"
Oct 31 12:43:10.821038 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:10.820998    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t" event={"ID":"87974688-1b9a-4778-8d77-158f5cdfa4aa","Type":"ContainerStarted","Data":"fbcfae886cc506f228fa7866b6cbc3f6e1994ab924ca24383307eca445712821"}
Oct 31 12:43:11.587379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:11.572489    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 12:43:11.587379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:11.572553    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 12:43:11.605294 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:11.601556    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:11.828648 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:11.828610    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t" event={"ID":"87974688-1b9a-4778-8d77-158f5cdfa4aa","Type":"ContainerStarted","Data":"2ff74dfede13b1229423c433b9e081d258f8dc0ac6c036c60185e4506eb20f26"}
Oct 31 12:43:11.828854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:11.828832    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:11.849597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:11.849489    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t" podStartSLOduration=2.394368871 podStartE2EDuration="2.849474633s" totalImagesPullingTime="455.105762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:11.849039757 +0000 UTC m=+916.589465975" watchObservedRunningTime="2026-10-31 12:43:11.849474633 +0000 UTC m=+916.589900849"
Oct 31 12:43:12.560379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:12.559570    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 12:43:12.560379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:12.559653    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 12:43:13.581380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:13.580786    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 12:43:13.581380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:13.580851    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 12:43:13.826435 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:13.826394    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"]
Oct 31 12:43:13.831043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:13.831018    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"
Oct 31 12:43:13.840931 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:13.840904    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-43c7f8da\""
Oct 31 12:43:13.841769 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:13.841751    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-43c7f8da-dockercfg-85gwm\""
Oct 31 12:43:13.842983 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:13.842958    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"]
Oct 31 12:43:13.888532 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:13.888482    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p8w8\" (UniqueName: \"kubernetes.io/projected/31aadf71-6831-4aee-99a5-b92f113979e8-kube-api-access-8p8w8\") pod \"payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv\" (UID: \"31aadf71-6831-4aee-99a5-b92f113979e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"
Oct 31 12:43:13.888809 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:13.888787    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31aadf71-6831-4aee-99a5-b92f113979e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv\" (UID: \"31aadf71-6831-4aee-99a5-b92f113979e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"
Oct 31 12:43:14.010377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:14.009603    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31aadf71-6831-4aee-99a5-b92f113979e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv\" (UID: \"31aadf71-6831-4aee-99a5-b92f113979e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"
Oct 31 12:43:14.010377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:14.009774    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p8w8\" (UniqueName: \"kubernetes.io/projected/31aadf71-6831-4aee-99a5-b92f113979e8-kube-api-access-8p8w8\") pod \"payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv\" (UID: \"31aadf71-6831-4aee-99a5-b92f113979e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"
Oct 31 12:43:14.028385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:14.011526    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31aadf71-6831-4aee-99a5-b92f113979e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv\" (UID: \"31aadf71-6831-4aee-99a5-b92f113979e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"
Oct 31 12:43:14.237492 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:14.235004    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p8w8\" (UniqueName: \"kubernetes.io/projected/31aadf71-6831-4aee-99a5-b92f113979e8-kube-api-access-8p8w8\") pod \"payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv\" (UID: \"31aadf71-6831-4aee-99a5-b92f113979e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"
Oct 31 12:43:14.449265 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:14.449227    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"
Oct 31 12:43:14.561432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:14.558347    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 12:43:14.561432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:14.558418    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 12:43:14.856429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:14.844158    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:15.214641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:15.214612    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"]
Oct 31 12:43:15.215329 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:43:15.215306    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31aadf71_6831_4aee_99a5_b92f113979e8.slice/crio-f56ebac9f78b4e5a1e2a15b94c1ce2bf148669ebd71800469408c688a4348f5a WatchSource:0}: Error finding container f56ebac9f78b4e5a1e2a15b94c1ce2bf148669ebd71800469408c688a4348f5a: Status 404 returned error can't find the container with id f56ebac9f78b4e5a1e2a15b94c1ce2bf148669ebd71800469408c688a4348f5a
Oct 31 12:43:15.581970 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:15.581932    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 12:43:15.582185 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:15.582001    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 12:43:15.899338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:15.899250    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv" event={"ID":"31aadf71-6831-4aee-99a5-b92f113979e8","Type":"ContainerStarted","Data":"bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986"}
Oct 31 12:43:15.899338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:15.899325    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"
Oct 31 12:43:15.899777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:15.899408    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv" event={"ID":"31aadf71-6831-4aee-99a5-b92f113979e8","Type":"ContainerStarted","Data":"f56ebac9f78b4e5a1e2a15b94c1ce2bf148669ebd71800469408c688a4348f5a"}
Oct 31 12:43:15.924425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:15.924373    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv" podStartSLOduration=2.924344822 podStartE2EDuration="2.924344822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:15.922000306 +0000 UTC m=+920.662426522" watchObservedRunningTime="2026-10-31 12:43:15.924344822 +0000 UTC m=+920.664771040"
Oct 31 12:43:16.558302 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:16.558268    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 12:43:16.558517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:16.558320    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 12:43:17.654176 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:17.654145    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:17.654627 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:17.654607    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:17.655344 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:17.655326    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:43:17.922434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:17.917675    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:22.602297 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:43:22.602106    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:43:23.137644 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.137607    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"]
Oct 31 12:43:23.142207 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.142178    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.148982 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.146851    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-43c7f8da-serving-cert\""
Oct 31 12:43:23.154629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.154200    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"]
Oct 31 12:43:23.236891 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.236855    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-maas-api-tls\") pod \"maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.237159 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.237139    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b87f42-178f-45b9-b976-8dbc89d87b04-openshift-service-ca\") pod \"maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.237209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.237194    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-metrics-certs\") pod \"maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.237286 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.237261    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgxgn\" (UniqueName: \"kubernetes.io/projected/b0b87f42-178f-45b9-b976-8dbc89d87b04-kube-api-access-tgxgn\") pod \"maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.377382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.339974    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b87f42-178f-45b9-b976-8dbc89d87b04-openshift-service-ca\") pod \"maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.377382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.340050    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-metrics-certs\") pod \"maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.377382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.340102    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgxgn\" (UniqueName: \"kubernetes.io/projected/b0b87f42-178f-45b9-b976-8dbc89d87b04-kube-api-access-tgxgn\") pod \"maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.377382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.340308    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-maas-api-tls\") pod \"maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.377382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.341957    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b87f42-178f-45b9-b976-8dbc89d87b04-openshift-service-ca\") pod \"maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.432281 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.424832    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-maas-api-tls\") pod \"maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.444622 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.436971    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-metrics-certs\") pod \"maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.448614 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.448581    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgxgn\" (UniqueName: \"kubernetes.io/projected/b0b87f42-178f-45b9-b976-8dbc89d87b04-kube-api-access-tgxgn\") pod \"maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.479386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.478459    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:23.937670 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:23.922536    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"]
Oct 31 12:43:24.207542 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:24.207443    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l" event={"ID":"b0b87f42-178f-45b9-b976-8dbc89d87b04","Type":"ContainerStarted","Data":"ac6221248b310cf7a5c82b62f5de87c5e3cb33c79591f5130e845564422ebb27"}
Oct 31 12:43:25.038775 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.038734    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"]
Oct 31 12:43:25.044443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.044420    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"
Oct 31 12:43:25.052003 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.051976    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"]
Oct 31 12:43:25.058700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.058678    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ed25a690-dockercfg-j6lg6\""
Oct 31 12:43:25.144379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.103606    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ed25a690\""
Oct 31 12:43:25.144379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.116403    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984090e7-bd56-48d7-933a-a610666c747e-plugins-config-volume\") pod \"payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8\" (UID: \"984090e7-bd56-48d7-933a-a610666c747e\") " pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"
Oct 31 12:43:25.144379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.116526    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c82cb\" (UniqueName: \"kubernetes.io/projected/984090e7-bd56-48d7-933a-a610666c747e-kube-api-access-c82cb\") pod \"payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8\" (UID: \"984090e7-bd56-48d7-933a-a610666c747e\") " pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"
Oct 31 12:43:25.241099 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.241051    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984090e7-bd56-48d7-933a-a610666c747e-plugins-config-volume\") pod \"payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8\" (UID: \"984090e7-bd56-48d7-933a-a610666c747e\") " pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"
Oct 31 12:43:25.241294 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.241242    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c82cb\" (UniqueName: \"kubernetes.io/projected/984090e7-bd56-48d7-933a-a610666c747e-kube-api-access-c82cb\") pod \"payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8\" (UID: \"984090e7-bd56-48d7-933a-a610666c747e\") " pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"
Oct 31 12:43:25.266374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.255667    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984090e7-bd56-48d7-933a-a610666c747e-plugins-config-volume\") pod \"payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8\" (UID: \"984090e7-bd56-48d7-933a-a610666c747e\") " pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"
Oct 31 12:43:25.295060 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.294952    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c82cb\" (UniqueName: \"kubernetes.io/projected/984090e7-bd56-48d7-933a-a610666c747e-kube-api-access-c82cb\") pod \"payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8\" (UID: \"984090e7-bd56-48d7-933a-a610666c747e\") " pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"
Oct 31 12:43:25.328490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.328412    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l" event={"ID":"b0b87f42-178f-45b9-b976-8dbc89d87b04","Type":"ContainerStarted","Data":"f0ca3a378cada7e3382b6c23dde6553c41bc765584f399eba197b346c44223ac"}
Oct 31 12:43:25.363392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.362797    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"
Oct 31 12:43:25.709161 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.708942    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l" podStartSLOduration=2.181897644 podStartE2EDuration="2.708923434s" totalImagesPullingTime="527.02579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:25.363698285 +0000 UTC m=+930.104124502" watchObservedRunningTime="2026-10-31 12:43:25.708923434 +0000 UTC m=+930.449349652"
Oct 31 12:43:25.712620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:25.712592    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"]
Oct 31 12:43:25.712846 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:43:25.712824    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984090e7_bd56_48d7_933a_a610666c747e.slice/crio-a40ecf0d6cc17f5a91d5ff0aec521051a1204c0f74a76b0b18624badbaf8a2ff WatchSource:0}: Error finding container a40ecf0d6cc17f5a91d5ff0aec521051a1204c0f74a76b0b18624badbaf8a2ff: Status 404 returned error can't find the container with id a40ecf0d6cc17f5a91d5ff0aec521051a1204c0f74a76b0b18624badbaf8a2ff
Oct 31 12:43:26.339808 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:26.339774    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8" event={"ID":"984090e7-bd56-48d7-933a-a610666c747e","Type":"ContainerStarted","Data":"98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35"}
Oct 31 12:43:26.340224 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:26.339832    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8" event={"ID":"984090e7-bd56-48d7-933a-a610666c747e","Type":"ContainerStarted","Data":"a40ecf0d6cc17f5a91d5ff0aec521051a1204c0f74a76b0b18624badbaf8a2ff"}
Oct 31 12:43:26.340224 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:26.339860    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"
Oct 31 12:43:26.340224 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:26.340133    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:26.366762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:26.366717    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8" podStartSLOduration=1.366703022 podStartE2EDuration="1.366703022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:26.359674738 +0000 UTC m=+931.100100955" watchObservedRunningTime="2026-10-31 12:43:26.366703022 +0000 UTC m=+931.107129238"
Oct 31 12:43:26.651442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:26.651365    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"]
Oct 31 12:43:26.911089 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:26.910996    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"
Oct 31 12:43:31.549021 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:31.548979    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:43:31.710406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:31.666457    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm" podUID="8403b6b0-9749-4b66-a651-f1db7c6b6e71" containerName="payload-pre-processing" containerID="cri-o://324ff06430cbdc421616c07ecfef1d1d4f525b8f20a74ca801014630920dbc48" gracePeriod=25
Oct 31 12:43:32.016378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.003539    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:43:32.016378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.004942    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:32.091438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.091389    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:32.092178 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.092145    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:43:32.110472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.105823    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:43:32.143559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.143493    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:32.160498 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.160448    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:32.236387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.232534    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:32.341377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.339884    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:43:32.359520 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.358445    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:32.374477 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:43:32.368659    5663 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8403b6b0_9749_4b66_a651_f1db7c6b6e71.slice/crio-324ff06430cbdc421616c07ecfef1d1d4f525b8f20a74ca801014630920dbc48.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:43:32.435377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.423691    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:43:32.444933 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.444888    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:32.569379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.554556    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:43:32.569379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.555932    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:32.569379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.556551    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:43:32.964964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.937461    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:32.964964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:32.937546    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:33.001643 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.000767    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:33.001643 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.000833    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:33.048539 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.048486    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:33.049172 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.049137    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:43:33.049588 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.049570    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:43:33.061504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.061476    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:43:33.140117 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.139006    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:43:33.218100 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.191800    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:43:33.218100 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.202267    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:43:33.296986 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.232738    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:33.296986 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.295821    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:43:33.339088 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.326408    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:43:33.354403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.340647    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:43:33.354403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.340738    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:43:33.354403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.340798    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:43:33.354403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.340851    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:43:33.354403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.340887    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:43:33.354403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.340938    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:43:33.377754 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.377537    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:43:33.416871 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.377473    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:33.430526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.430493    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:43:33.434378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.430786    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:43:33.597378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.571423    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:33.597378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.571724    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:33.597378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.579276    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:43:33.621406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.605727    5663 generic.go:356] "Generic (PLEG): container finished" podID="8403b6b0-9749-4b66-a651-f1db7c6b6e71" containerID="324ff06430cbdc421616c07ecfef1d1d4f525b8f20a74ca801014630920dbc48" exitCode=0
Oct 31 12:43:33.692378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.692321    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:33.703408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.692400    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:33.838588 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.838541    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:33.852812 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.852719    5663 patch_prober.go:29] interesting pod/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:33.853055 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:33.853019    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.005741 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.005703    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:34.006006 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.005979    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.056599 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.056543    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.090397 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.072453    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.147533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.147431    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.166205 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.166166    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm" event={"ID":"8403b6b0-9749-4b66-a651-f1db7c6b6e71","Type":"ContainerDied","Data":"324ff06430cbdc421616c07ecfef1d1d4f525b8f20a74ca801014630920dbc48"}
Oct 31 12:43:34.166525 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.166504    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:43:34.166930 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.166878    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:43:34.534249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.166764    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"
Oct 31 12:43:34.576258 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.223510    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:43:34.576799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.223545    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:43:34.587120 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.251449    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.604384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.604274    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:43:34.604384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.424104    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.604946 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.425457    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.604946 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.425505    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:43:34.605062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.574505    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 12:43:34.605684 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.574539    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:43:34.606028 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.585903    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.619123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.310696    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.696403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.660722    5663 patch_prober.go:29] interesting pod/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:34.696403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.660787    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.696403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.663177    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8403b6b0-9749-4b66-a651-f1db7c6b6e71-kube-api-access-z7j8g\" (UniqueName: \"kubernetes.io/projected/8403b6b0-9749-4b66-a651-f1db7c6b6e71-kube-api-access-z7j8g\") pod \"8403b6b0-9749-4b66-a651-f1db7c6b6e71\" (UID: \"8403b6b0-9749-4b66-a651-f1db7c6b6e71\") "
Oct 31 12:43:34.697325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.697280    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.697737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.697706    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8403b6b0-9749-4b66-a651-f1db7c6b6e71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8403b6b0-9749-4b66-a651-f1db7c6b6e71-plugins-config-volume\") pod \"8403b6b0-9749-4b66-a651-f1db7c6b6e71\" (UID: \"8403b6b0-9749-4b66-a651-f1db7c6b6e71\") "
Oct 31 12:43:34.704544 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.704508    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:34.711785 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.711758    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8403b6b0-9749-4b66-a651-f1db7c6b6e71-plugins-config-volume" pod "8403b6b0-9749-4b66-a651-f1db7c6b6e71" (UID: "8403b6b0-9749-4b66-a651-f1db7c6b6e71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:34.769416 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.721563    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8403b6b0-9749-4b66-a651-f1db7c6b6e71-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:34.853389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.850552    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8403b6b0-9749-4b66-a651-f1db7c6b6e71-kube-api-access-z7j8g" pod "8403b6b0-9749-4b66-a651-f1db7c6b6e71" (UID: "8403b6b0-9749-4b66-a651-f1db7c6b6e71"). InnerVolumeSpecName "kube-api-access-z7j8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:34.853389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.853388    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"
Oct 31 12:43:34.886379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.872556    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm" event={"ID":"8403b6b0-9749-4b66-a651-f1db7c6b6e71","Type":"ContainerDied","Data":"a7398309c36f8ab26dc26b90a5a7bac21da40f90801d1fa1256a63b5f6066447"}
Oct 31 12:43:34.886379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.872642    5663 scope.go:122] "RemoveContainer" containerID="324ff06430cbdc421616c07ecfef1d1d4f525b8f20a74ca801014630920dbc48"
Oct 31 12:43:34.953375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:34.952531    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7j8g\" (UniqueName: \"kubernetes.io/projected/8403b6b0-9749-4b66-a651-f1db7c6b6e71-kube-api-access-z7j8g\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:35.096448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.087515    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.096448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.088936    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:43:35.130578 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.130524    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.152491 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.152451    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:43:35.152764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.152732    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.173380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.171456    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Oct 31 12:43:35.173380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.171593    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Oct 31 12:43:35.589793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.589749    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.594481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.594444    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.634422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.602327    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.637711 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.607652    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.662375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.607680    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.662967 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.607702    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:43:35.683432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.607719    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:43:35.684067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.607738    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 12:43:35.711380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.607757    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.713146 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.625670    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:35.774659 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.772441    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"]
Oct 31 12:43:35.790390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:35.788311    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-59fd8a28-5f8bb644bc-6hwpm"]
Oct 31 12:43:36.034531 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.032127    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:43:36.096383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.090520    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:43:36.139408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.139370    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.182520 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.182472    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:43:36.575525 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.575472    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 31 12:43:36.672384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.614543    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:43:36.672384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.624070    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.672384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.624252    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.672384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.642713    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:43:36.672384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.643114    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.672384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.651566    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 31 12:43:36.672384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.667681    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.721467 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.701423    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.721787 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.706985    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:43:36.722474 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.713466    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:36.739476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:36.713507    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.009796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.007481    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.167381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.164338    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.187217 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.168115    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:43:37.187217 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.175686    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.254464 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.185829    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.254464 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.190276    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:43:37.394005 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.393976    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:43:37.402682 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.401215    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"
Oct 31 12:43:37.548431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.548399    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8403b6b0-9749-4b66-a651-f1db7c6b6e71" path="/var/lib/kubelet/pods/8403b6b0-9749-4b66-a651-f1db7c6b6e71/volumes"
Oct 31 12:43:37.615580 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.615536    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.656088 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.655979    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.666449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.666387    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.679827 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.673504    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.690533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.690493    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.707117 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.690738    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.720831 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.720802    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:43:37.741091 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.740876    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:43:37.752287 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.752244    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:37.773966 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.764686    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:43:37.773966 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.766768    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:43:37.808273 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.808234    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:43:37.844938 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.826650    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:43:37.951586 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:37.951500    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:43:38.024840 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.024525    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:43:38.061376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.055441    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"]
Oct 31 12:43:38.130244 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.130215    5663 cpu_manager.go:414] "containerMap: removing container" podUID="8403b6b0-9749-4b66-a651-f1db7c6b6e71" containerName="payload-pre-processing"
Oct 31 12:43:38.136114 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.136086    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8403b6b0-9749-4b66-a651-f1db7c6b6e71" containerName="payload-pre-processing"
Oct 31 12:43:38.137469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.137444    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="8403b6b0-9749-4b66-a651-f1db7c6b6e71" containerName="payload-pre-processing"
Oct 31 12:43:38.190534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.187679    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:38.318524 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.305801    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.440481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.440440    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.440668 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.440559    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.440668 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.440585    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2phz7\" (UniqueName: \"kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-kube-api-access-2phz7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.440668 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.440614    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.440839 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.440699    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.440894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.440835    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.440894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.440889    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.452216 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.452185    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"]
Oct 31 12:43:38.462940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.462901    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:43:38.464962 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.464940    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:43:38.465855 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.465836    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:43:38.470456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.470434    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:43:38.478761 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.478741    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:43:38.542581 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.542546    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.542774 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.542695    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.543194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.543164    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.543312 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.543208    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2phz7\" (UniqueName: \"kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-kube-api-access-2phz7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.543312 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.543246    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.543452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.543421    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.543514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.543496    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.543641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.543619    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.549327 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.549297    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.550159 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.550134    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.551767 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.551736    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.551946 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.551927    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.556906 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.556328    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.557930 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.557884    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2phz7\" (UniqueName: \"kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-kube-api-access-2phz7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jjk8\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.678695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.677632    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:38.999916 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:38.999884    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"]
Oct 31 12:43:39.001624 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:43:39.001584    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae87d4b1_ee30_4565_9c7d_1e8ecf251cc6.slice/crio-98028c1def929ece42368160fcedb51244f28fd1ad1acd6d4b22a1a5ea275d07 WatchSource:0}: Error finding container 98028c1def929ece42368160fcedb51244f28fd1ad1acd6d4b22a1a5ea275d07: Status 404 returned error can't find the container with id 98028c1def929ece42368160fcedb51244f28fd1ad1acd6d4b22a1a5ea275d07
Oct 31 12:43:39.004866 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:39.004828    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:39.004959 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:39.004897    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:43:39.567647 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:39.567607    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8" event={"ID":"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6","Type":"ContainerStarted","Data":"f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff"}
Oct 31 12:43:39.568008 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:39.567980    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:39.568164 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:39.568146    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8" event={"ID":"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6","Type":"ContainerStarted","Data":"98028c1def929ece42368160fcedb51244f28fd1ad1acd6d4b22a1a5ea275d07"}
Oct 31 12:43:39.619527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:39.612775    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8" podStartSLOduration=1.612754611 podStartE2EDuration="1.612754611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:43:39.609824663 +0000 UTC m=+944.350250881" watchObservedRunningTime="2026-10-31 12:43:39.612754611 +0000 UTC m=+944.353180828"
Oct 31 12:43:40.532130 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:40.532098    5663 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5jjk8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:43:40.532537 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:40.532147    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8" podUID="ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:43:41.507711 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:41.506684    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"]
Oct 31 12:43:41.507711 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:41.506921    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf" podUID="4bb42103-e803-4ed2-a606-2fb5c3dc3fef" containerName="maas-api" containerID="cri-o://6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195" gracePeriod=30
Oct 31 12:43:41.573643 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:41.573610    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf" podUID="4bb42103-e803-4ed2-a606-2fb5c3dc3fef" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.74:8443/health\": dial tcp 10.133.0.74:8443: connect: connection refused"
Oct 31 12:43:42.374090 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.374065    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:42.416815 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.416784    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-maas-api-tls\") pod \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") "
Oct 31 12:43:42.417026 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.417010    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-kube-api-access-fcczb\" (UniqueName: \"kubernetes.io/projected/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-kube-api-access-fcczb\") pod \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") "
Oct 31 12:43:42.417105 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.417090    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-metrics-certs\") pod \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") "
Oct 31 12:43:42.417181 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.417150    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-openshift-service-ca\") pod \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\" (UID: \"4bb42103-e803-4ed2-a606-2fb5c3dc3fef\") "
Oct 31 12:43:42.419411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.417608    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-openshift-service-ca" pod "4bb42103-e803-4ed2-a606-2fb5c3dc3fef" (UID: "4bb42103-e803-4ed2-a606-2fb5c3dc3fef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:42.419411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.418182    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:42.420672 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.420644    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-maas-api-tls" pod "4bb42103-e803-4ed2-a606-2fb5c3dc3fef" (UID: "4bb42103-e803-4ed2-a606-2fb5c3dc3fef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:42.420799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.420691    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-kube-api-access-fcczb" pod "4bb42103-e803-4ed2-a606-2fb5c3dc3fef" (UID: "4bb42103-e803-4ed2-a606-2fb5c3dc3fef"). InnerVolumeSpecName "kube-api-access-fcczb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:42.421907 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.421882    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-metrics-certs" pod "4bb42103-e803-4ed2-a606-2fb5c3dc3fef" (UID: "4bb42103-e803-4ed2-a606-2fb5c3dc3fef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:42.520639 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.520602    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcczb\" (UniqueName: \"kubernetes.io/projected/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-kube-api-access-fcczb\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:42.520639 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.520639    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:42.520836 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.520655    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb42103-e803-4ed2-a606-2fb5c3dc3fef-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:42.565622 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.565578    5663 generic.go:356] "Generic (PLEG): container finished" podID="4bb42103-e803-4ed2-a606-2fb5c3dc3fef" containerID="6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195" exitCode=0
Oct 31 12:43:42.565798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.565627    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"
Oct 31 12:43:42.565798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.565657    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf" event={"ID":"4bb42103-e803-4ed2-a606-2fb5c3dc3fef","Type":"ContainerDied","Data":"6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195"}
Oct 31 12:43:42.565798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.565717    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf" event={"ID":"4bb42103-e803-4ed2-a606-2fb5c3dc3fef","Type":"ContainerDied","Data":"d0b0dbe5396e6613ea936a490f2ed9d86f74accb4f97c55bd0587906ea90c537"}
Oct 31 12:43:42.565798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.565739    5663 scope.go:122] "RemoveContainer" containerID="6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195"
Oct 31 12:43:42.580436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.580403    5663 scope.go:122] "RemoveContainer" containerID="6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195"
Oct 31 12:43:42.580834 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:43:42.580723    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195\": container with ID starting with 6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195 not found: ID does not exist" containerID="6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195"
Oct 31 12:43:42.580834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.580756    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195"} err="failed to get container status \"6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195\": rpc error: code = NotFound desc = could not find container \"6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195\": container with ID starting with 6c7ad6d53b0245576aa00f7c4cb9f61f131567d3b0bafb24450a62afdcdb7195 not found: ID does not exist"
Oct 31 12:43:42.597481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.597413    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"]
Oct 31 12:43:42.600172 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:42.600148    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59fd8a28-7bbb68cc88-nmxxf"]
Oct 31 12:43:43.443220 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:43.443189    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb42103-e803-4ed2-a606-2fb5c3dc3fef" path="/var/lib/kubelet/pods/4bb42103-e803-4ed2-a606-2fb5c3dc3fef/volumes"
Oct 31 12:43:43.530898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:43.530869    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:43:46.135434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:46.135398    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"]
Oct 31 12:43:46.135908 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:46.135635    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq" podUID="f28622af-2097-40b7-bf72-01b39d2bdfd7" containerName="maas-api" containerID="cri-o://1d705e98d60caefcb7e518b74fe1b6e4b544386edd876ce1a432f13ed4386a90" gracePeriod=30
Oct 31 12:43:46.823400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:46.782710    5663 generic.go:356] "Generic (PLEG): container finished" podID="f28622af-2097-40b7-bf72-01b39d2bdfd7" containerID="1d705e98d60caefcb7e518b74fe1b6e4b544386edd876ce1a432f13ed4386a90" exitCode=0
Oct 31 12:43:46.823400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:46.782938    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq" event={"ID":"f28622af-2097-40b7-bf72-01b39d2bdfd7","Type":"ContainerDied","Data":"1d705e98d60caefcb7e518b74fe1b6e4b544386edd876ce1a432f13ed4386a90"}
Oct 31 12:43:47.106563 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.099855    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:47.121522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.120171    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-metrics-certs\") pod \"f28622af-2097-40b7-bf72-01b39d2bdfd7\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") "
Oct 31 12:43:47.121522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.120547    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-maas-api-tls\") pod \"f28622af-2097-40b7-bf72-01b39d2bdfd7\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") "
Oct 31 12:43:47.121522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.120742    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28622af-2097-40b7-bf72-01b39d2bdfd7-kube-api-access-8v2tx\" (UniqueName: \"kubernetes.io/projected/f28622af-2097-40b7-bf72-01b39d2bdfd7-kube-api-access-8v2tx\") pod \"f28622af-2097-40b7-bf72-01b39d2bdfd7\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") "
Oct 31 12:43:47.121522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.120809    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28622af-2097-40b7-bf72-01b39d2bdfd7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f28622af-2097-40b7-bf72-01b39d2bdfd7-openshift-service-ca\") pod \"f28622af-2097-40b7-bf72-01b39d2bdfd7\" (UID: \"f28622af-2097-40b7-bf72-01b39d2bdfd7\") "
Oct 31 12:43:47.150433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.121949    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28622af-2097-40b7-bf72-01b39d2bdfd7-openshift-service-ca" pod "f28622af-2097-40b7-bf72-01b39d2bdfd7" (UID: "f28622af-2097-40b7-bf72-01b39d2bdfd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:47.184581 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.182167    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-metrics-certs" pod "f28622af-2097-40b7-bf72-01b39d2bdfd7" (UID: "f28622af-2097-40b7-bf72-01b39d2bdfd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:47.184581 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.182730    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28622af-2097-40b7-bf72-01b39d2bdfd7-kube-api-access-8v2tx" pod "f28622af-2097-40b7-bf72-01b39d2bdfd7" (UID: "f28622af-2097-40b7-bf72-01b39d2bdfd7"). InnerVolumeSpecName "kube-api-access-8v2tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:47.201879 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.201821    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-maas-api-tls" pod "f28622af-2097-40b7-bf72-01b39d2bdfd7" (UID: "f28622af-2097-40b7-bf72-01b39d2bdfd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:47.229409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.225746    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:47.229409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.225794    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v2tx\" (UniqueName: \"kubernetes.io/projected/f28622af-2097-40b7-bf72-01b39d2bdfd7-kube-api-access-8v2tx\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:47.229409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.225821    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f28622af-2097-40b7-bf72-01b39d2bdfd7-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:47.229409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.225835    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f28622af-2097-40b7-bf72-01b39d2bdfd7-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:47.336379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.334583    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"]
Oct 31 12:43:47.336379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:47.334828    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t" podUID="87974688-1b9a-4778-8d77-158f5cdfa4aa" containerName="maas-api" containerID="cri-o://2ff74dfede13b1229423c433b9e081d258f8dc0ac6c036c60185e4506eb20f26" gracePeriod=30
Oct 31 12:43:48.335017 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.331893    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq" event={"ID":"f28622af-2097-40b7-bf72-01b39d2bdfd7","Type":"ContainerDied","Data":"157aee3886c99e509d2fdde5e40f852c376bb18e34d2832eb99a76c23bdc513d"}
Oct 31 12:43:48.335017 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.332014    5663 scope.go:122] "RemoveContainer" containerID="1d705e98d60caefcb7e518b74fe1b6e4b544386edd876ce1a432f13ed4386a90"
Oct 31 12:43:48.335017 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.332252    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"
Oct 31 12:43:48.355629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.343197    5663 generic.go:356] "Generic (PLEG): container finished" podID="87974688-1b9a-4778-8d77-158f5cdfa4aa" containerID="2ff74dfede13b1229423c433b9e081d258f8dc0ac6c036c60185e4506eb20f26" exitCode=0
Oct 31 12:43:48.359735 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.350274    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t" event={"ID":"87974688-1b9a-4778-8d77-158f5cdfa4aa","Type":"ContainerDied","Data":"2ff74dfede13b1229423c433b9e081d258f8dc0ac6c036c60185e4506eb20f26"}
Oct 31 12:43:48.380726 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.380684    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:48.389002 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.388966    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"]
Oct 31 12:43:48.393575 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.393544    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d5e8c3f2-56484b65c6-44jxq"]
Oct 31 12:43:48.735000 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.733554    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87974688-1b9a-4778-8d77-158f5cdfa4aa-kube-api-access-qqrjz\" (UniqueName: \"kubernetes.io/projected/87974688-1b9a-4778-8d77-158f5cdfa4aa-kube-api-access-qqrjz\") pod \"87974688-1b9a-4778-8d77-158f5cdfa4aa\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") "
Oct 31 12:43:48.735000 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.733613    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-maas-api-tls\") pod \"87974688-1b9a-4778-8d77-158f5cdfa4aa\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") "
Oct 31 12:43:48.735000 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.733668    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-metrics-certs\") pod \"87974688-1b9a-4778-8d77-158f5cdfa4aa\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") "
Oct 31 12:43:48.735000 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.734003    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87974688-1b9a-4778-8d77-158f5cdfa4aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87974688-1b9a-4778-8d77-158f5cdfa4aa-openshift-service-ca\") pod \"87974688-1b9a-4778-8d77-158f5cdfa4aa\" (UID: \"87974688-1b9a-4778-8d77-158f5cdfa4aa\") "
Oct 31 12:43:48.735858 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.735541    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87974688-1b9a-4778-8d77-158f5cdfa4aa-openshift-service-ca" pod "87974688-1b9a-4778-8d77-158f5cdfa4aa" (UID: "87974688-1b9a-4778-8d77-158f5cdfa4aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:48.846553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.841654    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87974688-1b9a-4778-8d77-158f5cdfa4aa-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:48.938517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.938462    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87974688-1b9a-4778-8d77-158f5cdfa4aa-kube-api-access-qqrjz" pod "87974688-1b9a-4778-8d77-158f5cdfa4aa" (UID: "87974688-1b9a-4778-8d77-158f5cdfa4aa"). InnerVolumeSpecName "kube-api-access-qqrjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:48.965578 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.962317    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqrjz\" (UniqueName: \"kubernetes.io/projected/87974688-1b9a-4778-8d77-158f5cdfa4aa-kube-api-access-qqrjz\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:48.989306 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.989037    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-maas-api-tls" pod "87974688-1b9a-4778-8d77-158f5cdfa4aa" (UID: "87974688-1b9a-4778-8d77-158f5cdfa4aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:48.989306 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:48.989129    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-metrics-certs" pod "87974688-1b9a-4778-8d77-158f5cdfa4aa" (UID: "87974688-1b9a-4778-8d77-158f5cdfa4aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:43:49.063657 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:49.063622    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:49.063657 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:49.063654    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87974688-1b9a-4778-8d77-158f5cdfa4aa-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:49.378582 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:49.378551    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"
Oct 31 12:43:49.378582 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:49.378571    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t" event={"ID":"87974688-1b9a-4778-8d77-158f5cdfa4aa","Type":"ContainerDied","Data":"fbcfae886cc506f228fa7866b6cbc3f6e1994ab924ca24383307eca445712821"}
Oct 31 12:43:49.379443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:49.378631    5663 scope.go:122] "RemoveContainer" containerID="2ff74dfede13b1229423c433b9e081d258f8dc0ac6c036c60185e4506eb20f26"
Oct 31 12:43:49.431769 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:49.431733    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"]
Oct 31 12:43:49.442160 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:49.442126    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28622af-2097-40b7-bf72-01b39d2bdfd7" path="/var/lib/kubelet/pods/f28622af-2097-40b7-bf72-01b39d2bdfd7/volumes"
Oct 31 12:43:49.461689 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:49.461651    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t"]
Oct 31 12:43:50.847374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:50.843455    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b849a4c4-c47b4c98-lj54t" podUID="87974688-1b9a-4778-8d77-158f5cdfa4aa" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.77:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:43:51.450242 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:51.450208    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87974688-1b9a-4778-8d77-158f5cdfa4aa" path="/var/lib/kubelet/pods/87974688-1b9a-4778-8d77-158f5cdfa4aa/volumes"
Oct 31 12:43:52.491574 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:43:52.491538    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:43:53.547678 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:53.547644    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"]
Oct 31 12:43:58.550500 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:58.550430    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv" podUID="31aadf71-6831-4aee-99a5-b92f113979e8" containerName="payload-pre-processing" containerID="cri-o://bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986" gracePeriod=25
Oct 31 12:43:59.003986 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.003950    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"
Oct 31 12:43:59.155431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.148904    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31aadf71-6831-4aee-99a5-b92f113979e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31aadf71-6831-4aee-99a5-b92f113979e8-plugins-config-volume\") pod \"31aadf71-6831-4aee-99a5-b92f113979e8\" (UID: \"31aadf71-6831-4aee-99a5-b92f113979e8\") "
Oct 31 12:43:59.155431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.149021    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31aadf71-6831-4aee-99a5-b92f113979e8-kube-api-access-8p8w8\" (UniqueName: \"kubernetes.io/projected/31aadf71-6831-4aee-99a5-b92f113979e8-kube-api-access-8p8w8\") pod \"31aadf71-6831-4aee-99a5-b92f113979e8\" (UID: \"31aadf71-6831-4aee-99a5-b92f113979e8\") "
Oct 31 12:43:59.198375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.194367    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31aadf71-6831-4aee-99a5-b92f113979e8-plugins-config-volume" pod "31aadf71-6831-4aee-99a5-b92f113979e8" (UID: "31aadf71-6831-4aee-99a5-b92f113979e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:43:59.198375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.195021    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31aadf71-6831-4aee-99a5-b92f113979e8-kube-api-access-8p8w8" pod "31aadf71-6831-4aee-99a5-b92f113979e8" (UID: "31aadf71-6831-4aee-99a5-b92f113979e8"). InnerVolumeSpecName "kube-api-access-8p8w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:43:59.262794 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.262745    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31aadf71-6831-4aee-99a5-b92f113979e8-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:59.263183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.263154    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p8w8\" (UniqueName: \"kubernetes.io/projected/31aadf71-6831-4aee-99a5-b92f113979e8-kube-api-access-8p8w8\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:43:59.720062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.719974    5663 generic.go:356] "Generic (PLEG): container finished" podID="31aadf71-6831-4aee-99a5-b92f113979e8" containerID="bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986" exitCode=0
Oct 31 12:43:59.720552 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.720103    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv" event={"ID":"31aadf71-6831-4aee-99a5-b92f113979e8","Type":"ContainerDied","Data":"bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986"}
Oct 31 12:43:59.720552 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.720204    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv" event={"ID":"31aadf71-6831-4aee-99a5-b92f113979e8","Type":"ContainerDied","Data":"f56ebac9f78b4e5a1e2a15b94c1ce2bf148669ebd71800469408c688a4348f5a"}
Oct 31 12:43:59.720552 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.720246    5663 scope.go:122] "RemoveContainer" containerID="bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986"
Oct 31 12:43:59.720552 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.720480    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"
Oct 31 12:43:59.738328 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.738282    5663 scope.go:122] "RemoveContainer" containerID="bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986"
Oct 31 12:43:59.738856 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:43:59.738816    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986\": container with ID starting with bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986 not found: ID does not exist" containerID="bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986"
Oct 31 12:43:59.739035 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.739000    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986"} err="failed to get container status \"bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986\": rpc error: code = NotFound desc = could not find container \"bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986\": container with ID starting with bea4ed17718097514379fb4b1927c5cbe764e020c501a78adb2e3f1d493d1986 not found: ID does not exist"
Oct 31 12:43:59.742647 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.742619    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"]
Oct 31 12:43:59.746876 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:43:59.746843    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-43c7f8da-5664ddcbb7-mz5pv"]
Oct 31 12:44:01.440700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:01.440668    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31aadf71-6831-4aee-99a5-b92f113979e8" path="/var/lib/kubelet/pods/31aadf71-6831-4aee-99a5-b92f113979e8/volumes"
Oct 31 12:44:01.745423 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:01.745298    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"]
Oct 31 12:44:01.745620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:01.745594    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l" podUID="b0b87f42-178f-45b9-b976-8dbc89d87b04" containerName="maas-api" containerID="cri-o://f0ca3a378cada7e3382b6c23dde6553c41bc765584f399eba197b346c44223ac" gracePeriod=30
Oct 31 12:44:02.418373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:02.418265    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l" podUID="b0b87f42-178f-45b9-b976-8dbc89d87b04" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.79:8443/health\": dial tcp 10.133.0.79:8443: connect: connection refused"
Oct 31 12:44:02.986997 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:02.986960    5663 generic.go:356] "Generic (PLEG): container finished" podID="b0b87f42-178f-45b9-b976-8dbc89d87b04" containerID="f0ca3a378cada7e3382b6c23dde6553c41bc765584f399eba197b346c44223ac" exitCode=0
Oct 31 12:44:02.987448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:02.987065    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l" event={"ID":"b0b87f42-178f-45b9-b976-8dbc89d87b04","Type":"ContainerDied","Data":"f0ca3a378cada7e3382b6c23dde6553c41bc765584f399eba197b346c44223ac"}
Oct 31 12:44:02.993934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:02.993912    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:44:03.112017 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:03.111975    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b87f42-178f-45b9-b976-8dbc89d87b04-kube-api-access-tgxgn\" (UniqueName: \"kubernetes.io/projected/b0b87f42-178f-45b9-b976-8dbc89d87b04-kube-api-access-tgxgn\") pod \"b0b87f42-178f-45b9-b976-8dbc89d87b04\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") "
Oct 31 12:44:03.112212 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:03.112113    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-metrics-certs\") pod \"b0b87f42-178f-45b9-b976-8dbc89d87b04\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") "
Oct 31 12:44:03.112279 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:03.112263    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b87f42-178f-45b9-b976-8dbc89d87b04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b87f42-178f-45b9-b976-8dbc89d87b04-openshift-service-ca\") pod \"b0b87f42-178f-45b9-b976-8dbc89d87b04\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") "
Oct 31 12:44:03.112333 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:03.112306    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-maas-api-tls\") pod \"b0b87f42-178f-45b9-b976-8dbc89d87b04\" (UID: \"b0b87f42-178f-45b9-b976-8dbc89d87b04\") "
Oct 31 12:44:03.112659 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:03.112631    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b87f42-178f-45b9-b976-8dbc89d87b04-openshift-service-ca" pod "b0b87f42-178f-45b9-b976-8dbc89d87b04" (UID: "b0b87f42-178f-45b9-b976-8dbc89d87b04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:44:03.113164 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:03.113137    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b87f42-178f-45b9-b976-8dbc89d87b04-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:03.114444 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:03.114411    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b87f42-178f-45b9-b976-8dbc89d87b04-kube-api-access-tgxgn" pod "b0b87f42-178f-45b9-b976-8dbc89d87b04" (UID: "b0b87f42-178f-45b9-b976-8dbc89d87b04"). InnerVolumeSpecName "kube-api-access-tgxgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:44:03.114547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:03.114482    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-metrics-certs" pod "b0b87f42-178f-45b9-b976-8dbc89d87b04" (UID: "b0b87f42-178f-45b9-b976-8dbc89d87b04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:44:03.114609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:03.114574    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-maas-api-tls" pod "b0b87f42-178f-45b9-b976-8dbc89d87b04" (UID: "b0b87f42-178f-45b9-b976-8dbc89d87b04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:44:03.215109 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:03.215021    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:03.215109 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:03.215047    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b87f42-178f-45b9-b976-8dbc89d87b04-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:03.215109 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:03.215060    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgxgn\" (UniqueName: \"kubernetes.io/projected/b0b87f42-178f-45b9-b976-8dbc89d87b04-kube-api-access-tgxgn\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:04.046374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:04.046315    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l" event={"ID":"b0b87f42-178f-45b9-b976-8dbc89d87b04","Type":"ContainerDied","Data":"ac6221248b310cf7a5c82b62f5de87c5e3cb33c79591f5130e845564422ebb27"}
Oct 31 12:44:04.046972 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:04.046950    5663 scope.go:122] "RemoveContainer" containerID="f0ca3a378cada7e3382b6c23dde6553c41bc765584f399eba197b346c44223ac"
Oct 31 12:44:04.047267 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:04.047246    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"
Oct 31 12:44:04.110451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:04.110411    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"]
Oct 31 12:44:04.120951 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:04.120902    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-43c7f8da-8fd467b56-xcl9l"]
Oct 31 12:44:05.439520 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:05.439491    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b87f42-178f-45b9-b976-8dbc89d87b04" path="/var/lib/kubelet/pods/b0b87f42-178f-45b9-b976-8dbc89d87b04/volumes"
Oct 31 12:44:15.119910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.119862    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"]
Oct 31 12:44:15.120838 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.120814    5663 cpu_manager.go:414] "containerMap: removing container" podUID="31aadf71-6831-4aee-99a5-b92f113979e8" containerName="payload-pre-processing"
Oct 31 12:44:15.120953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.120841    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31aadf71-6831-4aee-99a5-b92f113979e8" containerName="payload-pre-processing"
Oct 31 12:44:15.120953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.120872    5663 cpu_manager.go:414] "containerMap: removing container" podUID="4bb42103-e803-4ed2-a606-2fb5c3dc3fef" containerName="maas-api"
Oct 31 12:44:15.120953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.120881    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb42103-e803-4ed2-a606-2fb5c3dc3fef" containerName="maas-api"
Oct 31 12:44:15.120953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.120900    5663 cpu_manager.go:414] "containerMap: removing container" podUID="87974688-1b9a-4778-8d77-158f5cdfa4aa" containerName="maas-api"
Oct 31 12:44:15.120953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.120907    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87974688-1b9a-4778-8d77-158f5cdfa4aa" containerName="maas-api"
Oct 31 12:44:15.120953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.120922    5663 cpu_manager.go:414] "containerMap: removing container" podUID="b0b87f42-178f-45b9-b976-8dbc89d87b04" containerName="maas-api"
Oct 31 12:44:15.120953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.120929    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b87f42-178f-45b9-b976-8dbc89d87b04" containerName="maas-api"
Oct 31 12:44:15.121335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.120980    5663 cpu_manager.go:414] "containerMap: removing container" podUID="f28622af-2097-40b7-bf72-01b39d2bdfd7" containerName="maas-api"
Oct 31 12:44:15.121335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.120988    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28622af-2097-40b7-bf72-01b39d2bdfd7" containerName="maas-api"
Oct 31 12:44:15.121335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.121132    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28622af-2097-40b7-bf72-01b39d2bdfd7" containerName="maas-api"
Oct 31 12:44:15.121335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.121146    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="31aadf71-6831-4aee-99a5-b92f113979e8" containerName="payload-pre-processing"
Oct 31 12:44:15.121335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.121159    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb42103-e803-4ed2-a606-2fb5c3dc3fef" containerName="maas-api"
Oct 31 12:44:15.121335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.121168    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="87974688-1b9a-4778-8d77-158f5cdfa4aa" containerName="maas-api"
Oct 31 12:44:15.121335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.121177    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b87f42-178f-45b9-b976-8dbc89d87b04" containerName="maas-api"
Oct 31 12:44:15.125072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.125040    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.137716 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.137682    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"]
Oct 31 12:44:15.138690 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.138670    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-2d6201-openshift-default-dockercfg-xklfg\""
Oct 31 12:44:15.240379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.221324    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-data\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.240379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.221475    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-podinfo\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.240379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.221541    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-certs\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.240379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.221570    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc77r\" (UniqueName: \"kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-kube-api-access-qc77r\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.240379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.221611    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-token\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.240379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.221656    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istiod-ca-cert\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.240379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.221718    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-socket\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.240379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.221790    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-envoy\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.240379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.221881    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-credential-socket\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.327377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.325423    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-envoy\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.327377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.325550    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-credential-socket\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.327377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.325633    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-data\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.327377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.325752    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-podinfo\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.327377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.325820    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-certs\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.327377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.325852    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc77r\" (UniqueName: \"kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-kube-api-access-qc77r\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.327377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.325897    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-token\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.327377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.325946    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istiod-ca-cert\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.327377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.326011    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-socket\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.327377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.326523    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-socket\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.364382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.359774    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-certs\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.364382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.360063    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-credential-socket\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.364382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.360400    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-data\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.376039 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.369644    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istiod-ca-cert\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.434381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.431928    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-envoy\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.435030 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.434998    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc77r\" (UniqueName: \"kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-kube-api-access-qc77r\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.445139 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.445079    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-podinfo\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.475377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.456928    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-token\") pod \"e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") " pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:15.753813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:15.753716    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:16.190328 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:16.190292    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"]
Oct 31 12:44:16.191603 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:44:16.191556    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bdfa24_6e4e_4d2c_a4ac_2c515e104761.slice/crio-2c58c9310e0df5505771fc42da8cb5b1bc1df4741ec515784964cdd3c255092b WatchSource:0}: Error finding container 2c58c9310e0df5505771fc42da8cb5b1bc1df4741ec515784964cdd3c255092b: Status 404 returned error can't find the container with id 2c58c9310e0df5505771fc42da8cb5b1bc1df4741ec515784964cdd3c255092b
Oct 31 12:44:16.194435 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:16.194393    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:16.194563 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:16.194484    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:16.194563 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:16.194528    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:17.201863 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:17.201823    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts" event={"ID":"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761","Type":"ContainerStarted","Data":"0b3beb4ccacf597afcdb510af4d624269171c71e59558473a1285a91a3d29011"}
Oct 31 12:44:17.202309 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:17.201886    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts" event={"ID":"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761","Type":"ContainerStarted","Data":"2c58c9310e0df5505771fc42da8cb5b1bc1df4741ec515784964cdd3c255092b"}
Oct 31 12:44:17.755400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:17.755342    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:17.761139 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:17.761112    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:17.791658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:17.791608    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts" podStartSLOduration=2.791593267 podStartE2EDuration="2.791593267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:17.250816442 +0000 UTC m=+981.991242655" watchObservedRunningTime="2026-10-31 12:44:17.791593267 +0000 UTC m=+982.532019556"
Oct 31 12:44:18.208622 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:18.208587    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:18.209622 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:18.209600    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 12:44:21.668377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:21.665106    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:21.691378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:21.688447    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:22.081377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.080457    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:22.157379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.153675    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:22.228379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.224457    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:22.674514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.626425    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:22.674514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.626750    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:44:22.674514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.651469    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:22.674514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.658440    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:22.694698 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.679213    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"]
Oct 31 12:44:22.754011 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.695010    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:22.754011 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.698338    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:44:22.754011 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.699179    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:22.754011 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.714900    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:44:22.754011 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.715282    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:44:22.754011 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.726919    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:22.811403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.795299    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:44:22.811403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.795374    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:44:22.811403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.809783    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"
Oct 31 12:44:22.836273 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.836231    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-2d6201\""
Oct 31 12:44:22.839015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.836915    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-2d6201-dockercfg-rc7kp\""
Oct 31 12:44:22.840087 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.840053    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:44:22.854522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.854490    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ed33ef-f491-4560-b06f-7ae78765e930-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x\" (UID: \"b2ed33ef-f491-4560-b06f-7ae78765e930\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"
Oct 31 12:44:22.917294 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.899568    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh9zq\" (UniqueName: \"kubernetes.io/projected/b2ed33ef-f491-4560-b06f-7ae78765e930-kube-api-access-jh9zq\") pod \"payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x\" (UID: \"b2ed33ef-f491-4560-b06f-7ae78765e930\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"
Oct 31 12:44:22.965673 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:22.965554    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"]
Oct 31 12:44:23.146382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.023403    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ed33ef-f491-4560-b06f-7ae78765e930-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x\" (UID: \"b2ed33ef-f491-4560-b06f-7ae78765e930\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"
Oct 31 12:44:23.154403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.024404    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ed33ef-f491-4560-b06f-7ae78765e930-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x\" (UID: \"b2ed33ef-f491-4560-b06f-7ae78765e930\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"
Oct 31 12:44:23.154719 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.154681    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:23.162385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.158808    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:44:23.190754 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.190711    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh9zq\" (UniqueName: \"kubernetes.io/projected/b2ed33ef-f491-4560-b06f-7ae78765e930-kube-api-access-jh9zq\") pod \"payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x\" (UID: \"b2ed33ef-f491-4560-b06f-7ae78765e930\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"
Oct 31 12:44:23.225566 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.225468    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:23.234448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.226183    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:44:23.648454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.644432    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh9zq\" (UniqueName: \"kubernetes.io/projected/b2ed33ef-f491-4560-b06f-7ae78765e930-kube-api-access-jh9zq\") pod \"payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x\" (UID: \"b2ed33ef-f491-4560-b06f-7ae78765e930\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"
Oct 31 12:44:23.764411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.764293    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 12:44:23.806556 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.806514    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:44:23.812419 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.806898    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:23.828445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.828397    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:23.831214 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.831177    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:23.833138 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.833108    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:44:23.837580 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:44:23.837548    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:44:23.839149 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.839118    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:44:23.910468 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.910368    5663 patch_prober.go:29] interesting pod/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:44:23.910719 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.910689    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:23.975384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:23.974148    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:24.002121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.002082    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:44:24.002636 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.002601    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:24.015375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.002779    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Oct 31 12:44:24.015631 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.004038    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:44:24.020757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.004086    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"
Oct 31 12:44:24.141653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.137394    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:44:24.141653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.137454    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:44:24.141653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.137490    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:44:24.141653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.137522    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:44:24.141653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.137551    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:44:24.141653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.137579    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:44:24.141653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.137606    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:44:24.141653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.137633    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:44:24.141653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.137661    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:44:24.141653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.137688    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:44:24.200485 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.175307    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:24.250062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.250022    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:44:24.378935 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.378907    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:44:24.598245 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.598215    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:44:24.705841 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.705815    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:44:24.811463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.767676    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:24.816228 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.813944    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:44:24.840747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.840710    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:24.842373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.842316    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:44:24.849225 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:24.849156    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:25.007522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.000345    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:25.032478 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.032428    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:44:25.244570 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.218554    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Oct 31 12:44:25.289393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.286581    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:25.619649 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.608942    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:25.630722 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.630670    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:25.643757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.643716    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:44:25.739185 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.709232    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:25.786384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.780926    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x" event={"ID":"b2ed33ef-f491-4560-b06f-7ae78765e930","Type":"ContainerStarted","Data":"82c1ed97342a683970920d94cc01427f2c5b774a349d7de602bf69162cb4d369"}
Oct 31 12:44:25.851084 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.850741    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:44:25.852417 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.852388    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:25.853582 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.853558    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:25.891409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.868554    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:44:25.891409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.880252    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:44:25.891409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.884043    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:44:25.965526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.954822    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:44:26.005934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.978508    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-dswld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:44:26.005934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:25.978584    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" podUID="57ad591e-3b45-4312-8612-cf848dcf55e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:26.026114 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:26.011434    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"]
Oct 31 12:44:26.085478 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:26.085426    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:44:26.139641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:26.137625    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:44:26.241073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:26.241040    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:44:26.268446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:26.264064    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:44:26.297369 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:26.286460    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:44:26.542478 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:26.541514    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:44:26.737589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:26.737305    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x" event={"ID":"b2ed33ef-f491-4560-b06f-7ae78765e930","Type":"ContainerStarted","Data":"ef1bd38f6fa02b5e95ad6c08e6f62fca88bee25432dfac7d37fb042d86841461"}
Oct 31 12:44:26.737589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:26.737419    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"
Oct 31 12:44:26.760293 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:26.760234    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x" podStartSLOduration=4.760216637 podStartE2EDuration="4.760216637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:26.756654791 +0000 UTC m=+991.497081008" watchObservedRunningTime="2026-10-31 12:44:26.760216637 +0000 UTC m=+991.500642855"
Oct 31 12:44:33.596584 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.596328    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"]
Oct 31 12:44:33.654615 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.654137    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.662470 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.662442    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"]
Oct 31 12:44:33.695911 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.695878    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ed25a690-openshift-default-dockercfg-wwhr7\""
Oct 31 12:44:33.786137 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.786099    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-certs\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.786325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.786153    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-envoy\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.786325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.786291    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-data\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.786531 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.786330    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e193d74c-a488-4a7b-83d2-866bb9edc44f-istiod-ca-cert\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.786531 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.786429    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.786531 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.786490    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxbl\" (UniqueName: \"kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-kube-api-access-7zxbl\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.786687 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.786610    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-podinfo\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.786770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.786746    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-token\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.786881 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.786865    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-credential-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.888958 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.888865    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-credential-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.889127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.889028    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-certs\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.889127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.889059    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-envoy\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.889127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.889093    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-data\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.889127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.889117    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e193d74c-a488-4a7b-83d2-866bb9edc44f-istiod-ca-cert\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.889380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.889145    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.889380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.889174    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxbl\" (UniqueName: \"kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-kube-api-access-7zxbl\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.889380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.889268    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-podinfo\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.889380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.889332    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-certs\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.889598 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.889390    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-token\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.890063 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.890037    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.890063 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.890053    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e193d74c-a488-4a7b-83d2-866bb9edc44f-istiod-ca-cert\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.890332 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.890313    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-data\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.890531 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.890514    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-credential-socket\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.892637 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.892614    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-envoy\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.892759 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.892741    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-podinfo\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.907596 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.907563    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-token\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.908224 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.908199    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxbl\" (UniqueName: \"kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-kube-api-access-7zxbl\") pod \"e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") " pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:33.989625 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:33.988834    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:34.436414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.436387    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"]
Oct 31 12:44:34.437536 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:44:34.437513    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode193d74c_a488_4a7b_83d2_866bb9edc44f.slice/crio-4ac11618288cd9a850955301143a60ecb8d35e1451422e3f6cedbdfda37b0da6 WatchSource:0}: Error finding container 4ac11618288cd9a850955301143a60ecb8d35e1451422e3f6cedbdfda37b0da6: Status 404 returned error can't find the container with id 4ac11618288cd9a850955301143a60ecb8d35e1451422e3f6cedbdfda37b0da6
Oct 31 12:44:34.440644 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.440612    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:34.440746 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.440675    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:34.440746 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.440706    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:34.474418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.474385    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"]
Oct 31 12:44:34.483709 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.483682    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.492870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.492708    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"]
Oct 31 12:44:34.492870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.492830    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-62734c-openshift-default-dockercfg-lsm7c\""
Oct 31 12:44:34.616376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.605514    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-899pt\" (UniqueName: \"kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-kube-api-access-899pt\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.616376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.605600    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-envoy\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.616376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.605680    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342f09b7-245d-4235-84da-8625ac04f790-istiod-ca-cert\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.616376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.605784    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-socket\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.616376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.605829    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342f09b7-245d-4235-84da-8625ac04f790-istio-podinfo\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.616376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.605906    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-istio-token\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.616376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.605973    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-certs\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.616376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.606013    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-credential-socket\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.616376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.606049    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-data\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.712465    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-899pt\" (UniqueName: \"kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-kube-api-access-899pt\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.712552    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-envoy\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.712632    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342f09b7-245d-4235-84da-8625ac04f790-istiod-ca-cert\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.712706    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-socket\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.712749    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342f09b7-245d-4235-84da-8625ac04f790-istio-podinfo\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.712802    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-istio-token\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.712845    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-certs\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.712883    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-credential-socket\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.712922    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-data\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.713585    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-data\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.714836    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-certs\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.718511    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-credential-socket\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.719181    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342f09b7-245d-4235-84da-8625ac04f790-istiod-ca-cert\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.719445    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-socket\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.727432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.721279    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-envoy\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.729115 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.729062    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342f09b7-245d-4235-84da-8625ac04f790-istio-podinfo\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.749057 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.749023    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-899pt\" (UniqueName: \"kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-kube-api-access-899pt\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.757445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.756852    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-istio-token\") pod \"e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") " pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.810854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.810725    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:34.876184 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.874956    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" event={"ID":"e193d74c-a488-4a7b-83d2-866bb9edc44f","Type":"ContainerStarted","Data":"1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46"}
Oct 31 12:44:34.876184 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:34.875035    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" event={"ID":"e193d74c-a488-4a7b-83d2-866bb9edc44f","Type":"ContainerStarted","Data":"4ac11618288cd9a850955301143a60ecb8d35e1451422e3f6cedbdfda37b0da6"}
Oct 31 12:44:35.013420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.001429    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:35.013420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.002368    5663 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 12:44:35.013420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.002420    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 12:44:35.694827 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:44:35.694786    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342f09b7_245d_4235_84da_8625ac04f790.slice/crio-f164a21480ec5aae0af654d96f8ef1f3d5b4ff9c5b82b9b05a29cb7b320dd05e WatchSource:0}: Error finding container f164a21480ec5aae0af654d96f8ef1f3d5b4ff9c5b82b9b05a29cb7b320dd05e: Status 404 returned error can't find the container with id f164a21480ec5aae0af654d96f8ef1f3d5b4ff9c5b82b9b05a29cb7b320dd05e
Oct 31 12:44:35.697326 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.697267    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" podStartSLOduration=2.697246838 podStartE2EDuration="2.697246838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:34.949422865 +0000 UTC m=+999.689849083" watchObservedRunningTime="2026-10-31 12:44:35.697246838 +0000 UTC m=+1000.437673056"
Oct 31 12:44:35.698287 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.698245    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:35.698434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.698337    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:35.698434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.698400    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:44:35.699002 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.698978    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"]
Oct 31 12:44:35.900920 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.900882    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" event={"ID":"342f09b7-245d-4235-84da-8625ac04f790","Type":"ContainerStarted","Data":"cbec050c1d07b9c5f7c745d6de64289e0d797b2b0db9731ae7566049fed690ce"}
Oct 31 12:44:35.901208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.901183    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" event={"ID":"342f09b7-245d-4235-84da-8625ac04f790","Type":"ContainerStarted","Data":"f164a21480ec5aae0af654d96f8ef1f3d5b4ff9c5b82b9b05a29cb7b320dd05e"}
Oct 31 12:44:35.927772 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.927714    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" podStartSLOduration=1.9277000530000001 podStartE2EDuration="1.927700053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:35.923607426 +0000 UTC m=+1000.664033643" watchObservedRunningTime="2026-10-31 12:44:35.927700053 +0000 UTC m=+1000.668126269"
Oct 31 12:44:35.991209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.991125    5663 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 12:44:35.991209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:35.991179    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 12:44:36.811121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:36.811087    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:36.817059 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:36.817027    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:36.907707 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:36.907667    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:36.908840 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:36.908817    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:44:37.005959 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:37.005921    5663 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 12:44:37.006240 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:37.006208    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 12:44:37.991884 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:37.991850    5663 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 12:44:37.992284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:37.991916    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 12:44:38.786404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:38.783616    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"
Oct 31 12:44:38.991263 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:38.991227    5663 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 12:44:38.991469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:38.991290    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 12:44:40.093376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:40.067888    5663 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 12:44:40.116520 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:40.116477    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 12:44:40.991858 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:40.991820    5663 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 12:44:40.992024 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:40.991876    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 12:44:41.991923 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:41.991888    5663 patch_prober.go:29] interesting pod/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 12:44:41.992444 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:41.991946    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 12:44:43.070958 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:43.070927    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:43.074647 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:43.074622    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:43.078955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:43.078933    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:44:50.828010 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:50.827969    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7029-dc989b"]
Oct 31 12:44:50.833061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:50.833038    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-dc989b"
Oct 31 12:44:50.837283 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:50.837254    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7029-dc989b"]
Oct 31 12:44:50.854245 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:50.839375    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-klvx2\""
Oct 31 12:44:50.977111 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:50.977079    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l499\" (UniqueName: \"kubernetes.io/projected/d768bc5d-7447-4719-a2da-04ca7d58cd5f-kube-api-access-8l499\") pod \"test-curl-7029-dc989b\" (UID: \"d768bc5d-7447-4719-a2da-04ca7d58cd5f\") " pod="opendatahub/test-curl-7029-dc989b"
Oct 31 12:44:51.078663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:51.078577    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l499\" (UniqueName: \"kubernetes.io/projected/d768bc5d-7447-4719-a2da-04ca7d58cd5f-kube-api-access-8l499\") pod \"test-curl-7029-dc989b\" (UID: \"d768bc5d-7447-4719-a2da-04ca7d58cd5f\") " pod="opendatahub/test-curl-7029-dc989b"
Oct 31 12:44:51.090780 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:51.090748    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l499\" (UniqueName: \"kubernetes.io/projected/d768bc5d-7447-4719-a2da-04ca7d58cd5f-kube-api-access-8l499\") pod \"test-curl-7029-dc989b\" (UID: \"d768bc5d-7447-4719-a2da-04ca7d58cd5f\") " pod="opendatahub/test-curl-7029-dc989b"
Oct 31 12:44:51.280663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:51.280624    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-dc989b"
Oct 31 12:44:51.621265 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:51.621239    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7029-dc989b"]
Oct 31 12:44:51.622320 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:44:51.622294    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd768bc5d_7447_4719_a2da_04ca7d58cd5f.slice/crio-4d69567074ccb9bc64d12e0403f9ced5739ebe9cc01a51938cec7f59b8b40522 WatchSource:0}: Error finding container 4d69567074ccb9bc64d12e0403f9ced5739ebe9cc01a51938cec7f59b8b40522: Status 404 returned error can't find the container with id 4d69567074ccb9bc64d12e0403f9ced5739ebe9cc01a51938cec7f59b8b40522
Oct 31 12:44:52.348338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:52.348301    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7029-dc989b" event={"ID":"d768bc5d-7447-4719-a2da-04ca7d58cd5f","Type":"ContainerStarted","Data":"4d69567074ccb9bc64d12e0403f9ced5739ebe9cc01a51938cec7f59b8b40522"}
Oct 31 12:44:52.477024 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:44:52.476985    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:44:53.386383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:53.385977    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7029-dc989b" event={"ID":"d768bc5d-7447-4719-a2da-04ca7d58cd5f","Type":"ContainerStarted","Data":"aa1925df7809ade2daabc48ab8e646da22d2024f452b29ad4c605d5c90552eae"}
Oct 31 12:44:53.447844 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:53.445296    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7029-dc989b" podStartSLOduration=2.189307631 podStartE2EDuration="3.445275219s" totalImagesPullingTime="1.255967588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:53.41764644 +0000 UTC m=+1018.158072657" watchObservedRunningTime="2026-10-31 12:44:53.445275219 +0000 UTC m=+1018.185701438"
Oct 31 12:44:54.120398 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.119988    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7029-dc989b"]
Oct 31 12:44:54.121230 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.121200    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7029-dc989b"]
Oct 31 12:44:54.263579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.263547    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7029-08c6ba"]
Oct 31 12:44:54.264627 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.264600    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d768bc5d-7447-4719-a2da-04ca7d58cd5f" containerName="test-curl-7029-dc989b"
Oct 31 12:44:54.264627 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.264627    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d768bc5d-7447-4719-a2da-04ca7d58cd5f" containerName="test-curl-7029-dc989b"
Oct 31 12:44:54.264842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.264825    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d768bc5d-7447-4719-a2da-04ca7d58cd5f" containerName="test-curl-7029-dc989b"
Oct 31 12:44:54.271343 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.271316    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-08c6ba"
Oct 31 12:44:54.272766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.272021    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7029-08c6ba"]
Oct 31 12:44:54.298867 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.298836    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vdz\" (UniqueName: \"kubernetes.io/projected/9d0d745e-e42a-47a1-aff3-e8d172dbdfd0-kube-api-access-l9vdz\") pod \"test-curl-7029-08c6ba\" (UID: \"9d0d745e-e42a-47a1-aff3-e8d172dbdfd0\") " pod="opendatahub/test-curl-7029-08c6ba"
Oct 31 12:44:54.400546 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.400461    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vdz\" (UniqueName: \"kubernetes.io/projected/9d0d745e-e42a-47a1-aff3-e8d172dbdfd0-kube-api-access-l9vdz\") pod \"test-curl-7029-08c6ba\" (UID: \"9d0d745e-e42a-47a1-aff3-e8d172dbdfd0\") " pod="opendatahub/test-curl-7029-08c6ba"
Oct 31 12:44:54.412681 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.412653    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vdz\" (UniqueName: \"kubernetes.io/projected/9d0d745e-e42a-47a1-aff3-e8d172dbdfd0-kube-api-access-l9vdz\") pod \"test-curl-7029-08c6ba\" (UID: \"9d0d745e-e42a-47a1-aff3-e8d172dbdfd0\") " pod="opendatahub/test-curl-7029-08c6ba"
Oct 31 12:44:54.589978 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.589944    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-08c6ba"
Oct 31 12:44:54.947607 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:54.947583    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7029-08c6ba"]
Oct 31 12:44:54.950408 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:44:54.950326    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d0d745e_e42a_47a1_aff3_e8d172dbdfd0.slice/crio-1611109525e25572ec1998360659932a55b9ddd47c33317bbdf8decffa9ba015 WatchSource:0}: Error finding container 1611109525e25572ec1998360659932a55b9ddd47c33317bbdf8decffa9ba015: Status 404 returned error can't find the container with id 1611109525e25572ec1998360659932a55b9ddd47c33317bbdf8decffa9ba015
Oct 31 12:44:55.400116 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:55.400081    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7029-08c6ba" event={"ID":"9d0d745e-e42a-47a1-aff3-e8d172dbdfd0","Type":"ContainerStarted","Data":"78a0e2c2c89fc76bbc3632152563a3c424e58183f41cc93981e062adc1782ae6"}
Oct 31 12:44:55.400313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:55.400126    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7029-08c6ba" event={"ID":"9d0d745e-e42a-47a1-aff3-e8d172dbdfd0","Type":"ContainerStarted","Data":"1611109525e25572ec1998360659932a55b9ddd47c33317bbdf8decffa9ba015"}
Oct 31 12:44:55.400313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:55.400168    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7029-dc989b" podUID="d768bc5d-7447-4719-a2da-04ca7d58cd5f" containerName="test-curl-7029-dc989b" containerID="cri-o://aa1925df7809ade2daabc48ab8e646da22d2024f452b29ad4c605d5c90552eae" gracePeriod=2
Oct 31 12:44:55.402705 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:44:55.402681    5663 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7029-dc989b\" is forbidden: User \"system:node:ip-10-0-140-137.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-137.us-east-2.compute.internal' and this object" podUID="d768bc5d-7447-4719-a2da-04ca7d58cd5f" pod="opendatahub/test-curl-7029-dc989b"
Oct 31 12:44:55.420293 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:55.420235    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7029-08c6ba" podStartSLOduration=1.420222855 podStartE2EDuration="1.420222855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:55.419398631 +0000 UTC m=+1020.159824848" watchObservedRunningTime="2026-10-31 12:44:55.420222855 +0000 UTC m=+1020.160649074"
Oct 31 12:44:55.421691 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:44:55.421664    5663 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7029-dc989b\" is forbidden: User \"system:node:ip-10-0-140-137.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-137.us-east-2.compute.internal' and this object" podUID="d768bc5d-7447-4719-a2da-04ca7d58cd5f" pod="opendatahub/test-curl-7029-dc989b"
Oct 31 12:44:55.438001 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:44:55.437972    5663 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7029-dc989b\" is forbidden: User \"system:node:ip-10-0-140-137.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-137.us-east-2.compute.internal' and this object" podUID="d768bc5d-7447-4719-a2da-04ca7d58cd5f" pod="opendatahub/test-curl-7029-dc989b"
Oct 31 12:44:56.149157 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.149126    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7029-08c6ba"]
Oct 31 12:44:56.155506 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.155474    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7029-08c6ba"]
Oct 31 12:44:56.333065 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.333028    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7029-ad8af6"]
Oct 31 12:44:56.352789 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.352753    5663 cpu_manager.go:414] "containerMap: removing container" podUID="9d0d745e-e42a-47a1-aff3-e8d172dbdfd0" containerName="test-curl-7029-08c6ba"
Oct 31 12:44:56.353014 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.352996    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0d745e-e42a-47a1-aff3-e8d172dbdfd0" containerName="test-curl-7029-08c6ba"
Oct 31 12:44:56.353391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.353372    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0d745e-e42a-47a1-aff3-e8d172dbdfd0" containerName="test-curl-7029-08c6ba"
Oct 31 12:44:56.362523 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.362494    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-ad8af6"
Oct 31 12:44:56.372944 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.372915    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7029-ad8af6"]
Oct 31 12:44:56.492175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.492088    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfmf2\" (UniqueName: \"kubernetes.io/projected/2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4-kube-api-access-bfmf2\") pod \"test-curl-7029-ad8af6\" (UID: \"2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4\") " pod="opendatahub/test-curl-7029-ad8af6"
Oct 31 12:44:56.593486 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.593447    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfmf2\" (UniqueName: \"kubernetes.io/projected/2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4-kube-api-access-bfmf2\") pod \"test-curl-7029-ad8af6\" (UID: \"2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4\") " pod="opendatahub/test-curl-7029-ad8af6"
Oct 31 12:44:56.605553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.605523    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfmf2\" (UniqueName: \"kubernetes.io/projected/2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4-kube-api-access-bfmf2\") pod \"test-curl-7029-ad8af6\" (UID: \"2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4\") " pod="opendatahub/test-curl-7029-ad8af6"
Oct 31 12:44:56.699256 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.699229    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-ad8af6"
Oct 31 12:44:56.986423 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:56.986395    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7029-ad8af6"]
Oct 31 12:44:57.418795 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:57.418763    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7029-ad8af6" event={"ID":"2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4","Type":"ContainerStarted","Data":"cad9d30bc8f6616c340b695ffa08c0dccb156d17438a2046f910264796acd6bf"}
Oct 31 12:44:57.418939 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:57.418811    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7029-ad8af6" event={"ID":"2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4","Type":"ContainerStarted","Data":"4b201d6d9404fffce9e28c22905883fb493b7a951d17d7b186048e696b9c2a1a"}
Oct 31 12:44:57.419076 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:57.418994    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7029-08c6ba" podUID="9d0d745e-e42a-47a1-aff3-e8d172dbdfd0" containerName="test-curl-7029-08c6ba" containerID="cri-o://78a0e2c2c89fc76bbc3632152563a3c424e58183f41cc93981e062adc1782ae6" gracePeriod=2
Oct 31 12:44:57.437654 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:57.437608    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7029-ad8af6" podStartSLOduration=1.437581578 podStartE2EDuration="1.437581578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:57.436307741 +0000 UTC m=+1022.176733953" watchObservedRunningTime="2026-10-31 12:44:57.437581578 +0000 UTC m=+1022.178007794"
Oct 31 12:44:57.438863 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:44:57.438835    5663 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7029-08c6ba\" is forbidden: User \"system:node:ip-10-0-140-137.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-137.us-east-2.compute.internal' and this object" podUID="9d0d745e-e42a-47a1-aff3-e8d172dbdfd0" pod="opendatahub/test-curl-7029-08c6ba"
Oct 31 12:44:57.560314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:57.560287    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-dc989b"
Oct 31 12:44:57.610204 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:57.610171    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d768bc5d-7447-4719-a2da-04ca7d58cd5f-kube-api-access-8l499\" (UniqueName: \"kubernetes.io/projected/d768bc5d-7447-4719-a2da-04ca7d58cd5f-kube-api-access-8l499\") pod \"d768bc5d-7447-4719-a2da-04ca7d58cd5f\" (UID: \"d768bc5d-7447-4719-a2da-04ca7d58cd5f\") "
Oct 31 12:44:57.612255 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:57.612226    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d768bc5d-7447-4719-a2da-04ca7d58cd5f-kube-api-access-8l499" pod "d768bc5d-7447-4719-a2da-04ca7d58cd5f" (UID: "d768bc5d-7447-4719-a2da-04ca7d58cd5f"). InnerVolumeSpecName "kube-api-access-8l499". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:44:57.712787 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:57.712711    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l499\" (UniqueName: \"kubernetes.io/projected/d768bc5d-7447-4719-a2da-04ca7d58cd5f-kube-api-access-8l499\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:44:58.250293 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.250251    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7029-ad8af6"]
Oct 31 12:44:58.254530 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.254264    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7029-ad8af6"]
Oct 31 12:44:58.390561 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.390530    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7029-87fbbc"]
Oct 31 12:44:58.391176 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.391159    5663 cpu_manager.go:414] "containerMap: removing container" podUID="2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4" containerName="test-curl-7029-ad8af6"
Oct 31 12:44:58.391221 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.391179    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4" containerName="test-curl-7029-ad8af6"
Oct 31 12:44:58.391307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.391298    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4" containerName="test-curl-7029-ad8af6"
Oct 31 12:44:58.394950 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.394931    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-87fbbc"
Oct 31 12:44:58.402414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.401763    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7029-87fbbc"]
Oct 31 12:44:58.428072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.428040    5663 generic.go:356] "Generic (PLEG): container finished" podID="d768bc5d-7447-4719-a2da-04ca7d58cd5f" containerID="aa1925df7809ade2daabc48ab8e646da22d2024f452b29ad4c605d5c90552eae" exitCode=137
Oct 31 12:44:58.428216 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.428159    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-dc989b"
Oct 31 12:44:58.428216 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.428193    5663 scope.go:122] "RemoveContainer" containerID="aa1925df7809ade2daabc48ab8e646da22d2024f452b29ad4c605d5c90552eae"
Oct 31 12:44:58.437776 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.437748    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57mss\" (UniqueName: \"kubernetes.io/projected/82ea9df7-76a8-4c74-90ac-4d5d5d312644-kube-api-access-57mss\") pod \"test-curl-7029-87fbbc\" (UID: \"82ea9df7-76a8-4c74-90ac-4d5d5d312644\") " pod="opendatahub/test-curl-7029-87fbbc"
Oct 31 12:44:58.441390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.441331    5663 scope.go:122] "RemoveContainer" containerID="aa1925df7809ade2daabc48ab8e646da22d2024f452b29ad4c605d5c90552eae"
Oct 31 12:44:58.441685 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:44:58.441659    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1925df7809ade2daabc48ab8e646da22d2024f452b29ad4c605d5c90552eae\": container with ID starting with aa1925df7809ade2daabc48ab8e646da22d2024f452b29ad4c605d5c90552eae not found: ID does not exist" containerID="aa1925df7809ade2daabc48ab8e646da22d2024f452b29ad4c605d5c90552eae"
Oct 31 12:44:58.441769 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.441698    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1925df7809ade2daabc48ab8e646da22d2024f452b29ad4c605d5c90552eae"} err="failed to get container status \"aa1925df7809ade2daabc48ab8e646da22d2024f452b29ad4c605d5c90552eae\": rpc error: code = NotFound desc = could not find container \"aa1925df7809ade2daabc48ab8e646da22d2024f452b29ad4c605d5c90552eae\": container with ID starting with aa1925df7809ade2daabc48ab8e646da22d2024f452b29ad4c605d5c90552eae not found: ID does not exist"
Oct 31 12:44:58.448294 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:44:58.448263    5663 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7029-dc989b\" is forbidden: User \"system:node:ip-10-0-140-137.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-137.us-east-2.compute.internal' and this object" podUID="d768bc5d-7447-4719-a2da-04ca7d58cd5f" pod="opendatahub/test-curl-7029-dc989b"
Oct 31 12:44:58.539258 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.539181    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57mss\" (UniqueName: \"kubernetes.io/projected/82ea9df7-76a8-4c74-90ac-4d5d5d312644-kube-api-access-57mss\") pod \"test-curl-7029-87fbbc\" (UID: \"82ea9df7-76a8-4c74-90ac-4d5d5d312644\") " pod="opendatahub/test-curl-7029-87fbbc"
Oct 31 12:44:58.551325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.551300    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57mss\" (UniqueName: \"kubernetes.io/projected/82ea9df7-76a8-4c74-90ac-4d5d5d312644-kube-api-access-57mss\") pod \"test-curl-7029-87fbbc\" (UID: \"82ea9df7-76a8-4c74-90ac-4d5d5d312644\") " pod="opendatahub/test-curl-7029-87fbbc"
Oct 31 12:44:58.714925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:58.714893    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-87fbbc"
Oct 31 12:44:59.026097 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:59.025919    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7029-87fbbc"]
Oct 31 12:44:59.029287 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:44:59.029251    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82ea9df7_76a8_4c74_90ac_4d5d5d312644.slice/crio-d935f7f07913e57f06980a033e592a89c8b0c9fa9e2334eaf081c2b64db3c51c WatchSource:0}: Error finding container d935f7f07913e57f06980a033e592a89c8b0c9fa9e2334eaf081c2b64db3c51c: Status 404 returned error can't find the container with id d935f7f07913e57f06980a033e592a89c8b0c9fa9e2334eaf081c2b64db3c51c
Oct 31 12:44:59.506190 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:59.488740    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d768bc5d-7447-4719-a2da-04ca7d58cd5f" path="/var/lib/kubelet/pods/d768bc5d-7447-4719-a2da-04ca7d58cd5f/volumes"
Oct 31 12:44:59.521376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:59.508688    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7029-87fbbc" event={"ID":"82ea9df7-76a8-4c74-90ac-4d5d5d312644","Type":"ContainerStarted","Data":"5d0b9a1f06a60927d92dc3dbfce955a1b822b82a941a9168d14688bcc02f0a1c"}
Oct 31 12:44:59.521376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:59.508741    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7029-87fbbc" event={"ID":"82ea9df7-76a8-4c74-90ac-4d5d5d312644","Type":"ContainerStarted","Data":"d935f7f07913e57f06980a033e592a89c8b0c9fa9e2334eaf081c2b64db3c51c"}
Oct 31 12:44:59.521376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:59.520679    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7029-ad8af6" podUID="2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4" containerName="test-curl-7029-ad8af6" containerID="cri-o://cad9d30bc8f6616c340b695ffa08c0dccb156d17438a2046f910264796acd6bf" gracePeriod=2
Oct 31 12:44:59.563715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:59.552336    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7029-87fbbc" podStartSLOduration=1.552311167 podStartE2EDuration="1.552311167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:44:59.531004011 +0000 UTC m=+1024.271430230" watchObservedRunningTime="2026-10-31 12:44:59.552311167 +0000 UTC m=+1024.292737384"
Oct 31 12:44:59.765197 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:59.765168    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-08c6ba"
Oct 31 12:44:59.916465 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:59.915061    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0d745e-e42a-47a1-aff3-e8d172dbdfd0-kube-api-access-l9vdz\" (UniqueName: \"kubernetes.io/projected/9d0d745e-e42a-47a1-aff3-e8d172dbdfd0-kube-api-access-l9vdz\") pod \"9d0d745e-e42a-47a1-aff3-e8d172dbdfd0\" (UID: \"9d0d745e-e42a-47a1-aff3-e8d172dbdfd0\") "
Oct 31 12:44:59.945562 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:44:59.925716    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0d745e-e42a-47a1-aff3-e8d172dbdfd0-kube-api-access-l9vdz" pod "9d0d745e-e42a-47a1-aff3-e8d172dbdfd0" (UID: "9d0d745e-e42a-47a1-aff3-e8d172dbdfd0"). InnerVolumeSpecName "kube-api-access-l9vdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:00.018299 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.018167    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9vdz\" (UniqueName: \"kubernetes.io/projected/9d0d745e-e42a-47a1-aff3-e8d172dbdfd0-kube-api-access-l9vdz\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:00.156258 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.156223    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"]
Oct 31 12:45:00.160631 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.160609    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"
Oct 31 12:45:00.172440 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.171642    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:45:00.179699 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.179467    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"]
Oct 31 12:45:00.186327 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.186301    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"]
Oct 31 12:45:00.192309 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.191682    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"
Oct 31 12:45:00.198294 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.197653    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"]
Oct 31 12:45:00.222535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.222508    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"
Oct 31 12:45:00.245407 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.240712    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"]
Oct 31 12:45:00.258682 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.258552    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"]
Oct 31 12:45:00.258682 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.258608    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"]
Oct 31 12:45:00.258682 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.258636    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"]
Oct 31 12:45:00.258963 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.258731    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"
Oct 31 12:45:00.273488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.273462    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"]
Oct 31 12:45:00.278557 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.278535    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"]
Oct 31 12:45:00.278807 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.278793    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"
Oct 31 12:45:00.313272 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.313227    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7029-87fbbc"]
Oct 31 12:45:00.317670 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.317641    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7029-87fbbc"]
Oct 31 12:45:00.323442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.323391    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2g6\" (UniqueName: \"kubernetes.io/projected/961f2b49-e1fc-4d28-93f5-880f791be763-kube-api-access-vx2g6\") pod \"maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t\" (UID: \"961f2b49-e1fc-4d28-93f5-880f791be763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"
Oct 31 12:45:00.323711 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.323690    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp\" (UID: \"f98924ce-f10b-45fd-8a83-9f8ebfa424f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"
Oct 31 12:45:00.323918 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.323901    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961f2b49-e1fc-4d28-93f5-880f791be763-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t\" (UID: \"961f2b49-e1fc-4d28-93f5-880f791be763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"
Oct 31 12:45:00.324063 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.324046    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwtr\" (UniqueName: \"kubernetes.io/projected/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-kube-api-access-cfwtr\") pod \"maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp\" (UID: \"f98924ce-f10b-45fd-8a83-9f8ebfa424f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"
Oct 31 12:45:00.324488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.324467    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6d164c-805d-4977-b40a-20cc011edfb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl\" (UID: \"1c6d164c-805d-4977-b40a-20cc011edfb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"
Oct 31 12:45:00.324621 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.324602    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rkd7\" (UniqueName: \"kubernetes.io/projected/1c6d164c-805d-4977-b40a-20cc011edfb3-kube-api-access-5rkd7\") pod \"maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl\" (UID: \"1c6d164c-805d-4977-b40a-20cc011edfb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"
Oct 31 12:45:00.425962 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.425927    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6d164c-805d-4977-b40a-20cc011edfb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl\" (UID: \"1c6d164c-805d-4977-b40a-20cc011edfb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"
Oct 31 12:45:00.426142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.425976    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rkd7\" (UniqueName: \"kubernetes.io/projected/1c6d164c-805d-4977-b40a-20cc011edfb3-kube-api-access-5rkd7\") pod \"maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl\" (UID: \"1c6d164c-805d-4977-b40a-20cc011edfb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"
Oct 31 12:45:00.426142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.426102    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26790f61-8a65-4d3a-8927-c6bb7406b216-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck\" (UID: \"26790f61-8a65-4d3a-8927-c6bb7406b216\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"
Oct 31 12:45:00.426313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.426157    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2g6\" (UniqueName: \"kubernetes.io/projected/961f2b49-e1fc-4d28-93f5-880f791be763-kube-api-access-vx2g6\") pod \"maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t\" (UID: \"961f2b49-e1fc-4d28-93f5-880f791be763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"
Oct 31 12:45:00.426313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.426226    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crwtd\" (UniqueName: \"kubernetes.io/projected/26790f61-8a65-4d3a-8927-c6bb7406b216-kube-api-access-crwtd\") pod \"maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck\" (UID: \"26790f61-8a65-4d3a-8927-c6bb7406b216\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"
Oct 31 12:45:00.426313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.426303    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f23220-3a1e-457b-9f44-8e64fc4e131e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf\" (UID: \"87f23220-3a1e-457b-9f44-8e64fc4e131e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"
Oct 31 12:45:00.426498 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.426448    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp\" (UID: \"f98924ce-f10b-45fd-8a83-9f8ebfa424f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"
Oct 31 12:45:00.426557 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.426538    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961f2b49-e1fc-4d28-93f5-880f791be763-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t\" (UID: \"961f2b49-e1fc-4d28-93f5-880f791be763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"
Oct 31 12:45:00.426613 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.426574    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwtr\" (UniqueName: \"kubernetes.io/projected/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-kube-api-access-cfwtr\") pod \"maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp\" (UID: \"f98924ce-f10b-45fd-8a83-9f8ebfa424f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"
Oct 31 12:45:00.426667 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.426620    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6d164c-805d-4977-b40a-20cc011edfb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl\" (UID: \"1c6d164c-805d-4977-b40a-20cc011edfb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"
Oct 31 12:45:00.426725 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.426710    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ckwn\" (UniqueName: \"kubernetes.io/projected/87f23220-3a1e-457b-9f44-8e64fc4e131e-kube-api-access-4ckwn\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf\" (UID: \"87f23220-3a1e-457b-9f44-8e64fc4e131e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"
Oct 31 12:45:00.427223 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.427196    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp\" (UID: \"f98924ce-f10b-45fd-8a83-9f8ebfa424f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"
Oct 31 12:45:00.427325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.427204    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961f2b49-e1fc-4d28-93f5-880f791be763-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t\" (UID: \"961f2b49-e1fc-4d28-93f5-880f791be763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"
Oct 31 12:45:00.442004 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.441965    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwtr\" (UniqueName: \"kubernetes.io/projected/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-kube-api-access-cfwtr\") pod \"maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp\" (UID: \"f98924ce-f10b-45fd-8a83-9f8ebfa424f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"
Oct 31 12:45:00.442396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.442342    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rkd7\" (UniqueName: \"kubernetes.io/projected/1c6d164c-805d-4977-b40a-20cc011edfb3-kube-api-access-5rkd7\") pod \"maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl\" (UID: \"1c6d164c-805d-4977-b40a-20cc011edfb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"
Oct 31 12:45:00.442573 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.442552    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2g6\" (UniqueName: \"kubernetes.io/projected/961f2b49-e1fc-4d28-93f5-880f791be763-kube-api-access-vx2g6\") pod \"maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t\" (UID: \"961f2b49-e1fc-4d28-93f5-880f791be763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"
Oct 31 12:45:00.487058 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.487024    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"
Oct 31 12:45:00.524605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.524509    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"
Oct 31 12:45:00.528162 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.528126    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26790f61-8a65-4d3a-8927-c6bb7406b216-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck\" (UID: \"26790f61-8a65-4d3a-8927-c6bb7406b216\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"
Oct 31 12:45:00.528314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.528182    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crwtd\" (UniqueName: \"kubernetes.io/projected/26790f61-8a65-4d3a-8927-c6bb7406b216-kube-api-access-crwtd\") pod \"maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck\" (UID: \"26790f61-8a65-4d3a-8927-c6bb7406b216\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"
Oct 31 12:45:00.528314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.528235    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f23220-3a1e-457b-9f44-8e64fc4e131e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf\" (UID: \"87f23220-3a1e-457b-9f44-8e64fc4e131e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"
Oct 31 12:45:00.528509 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.528486    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ckwn\" (UniqueName: \"kubernetes.io/projected/87f23220-3a1e-457b-9f44-8e64fc4e131e-kube-api-access-4ckwn\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf\" (UID: \"87f23220-3a1e-457b-9f44-8e64fc4e131e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"
Oct 31 12:45:00.528854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.528830    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26790f61-8a65-4d3a-8927-c6bb7406b216-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck\" (UID: \"26790f61-8a65-4d3a-8927-c6bb7406b216\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"
Oct 31 12:45:00.529210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.529108    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f23220-3a1e-457b-9f44-8e64fc4e131e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf\" (UID: \"87f23220-3a1e-457b-9f44-8e64fc4e131e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"
Oct 31 12:45:00.530641 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.530616    5663 generic.go:356] "Generic (PLEG): container finished" podID="9d0d745e-e42a-47a1-aff3-e8d172dbdfd0" containerID="78a0e2c2c89fc76bbc3632152563a3c424e58183f41cc93981e062adc1782ae6" exitCode=137
Oct 31 12:45:00.530730 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.530720    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-08c6ba"
Oct 31 12:45:00.530790 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.530756    5663 scope.go:122] "RemoveContainer" containerID="78a0e2c2c89fc76bbc3632152563a3c424e58183f41cc93981e062adc1782ae6"
Oct 31 12:45:00.558747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.544446    5663 scope.go:122] "RemoveContainer" containerID="78a0e2c2c89fc76bbc3632152563a3c424e58183f41cc93981e062adc1782ae6"
Oct 31 12:45:00.558747 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:00.544819    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a0e2c2c89fc76bbc3632152563a3c424e58183f41cc93981e062adc1782ae6\": container with ID starting with 78a0e2c2c89fc76bbc3632152563a3c424e58183f41cc93981e062adc1782ae6 not found: ID does not exist" containerID="78a0e2c2c89fc76bbc3632152563a3c424e58183f41cc93981e062adc1782ae6"
Oct 31 12:45:00.558747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.544871    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a0e2c2c89fc76bbc3632152563a3c424e58183f41cc93981e062adc1782ae6"} err="failed to get container status \"78a0e2c2c89fc76bbc3632152563a3c424e58183f41cc93981e062adc1782ae6\": rpc error: code = NotFound desc = could not find container \"78a0e2c2c89fc76bbc3632152563a3c424e58183f41cc93981e062adc1782ae6\": container with ID starting with 78a0e2c2c89fc76bbc3632152563a3c424e58183f41cc93981e062adc1782ae6 not found: ID does not exist"
Oct 31 12:45:00.558747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.546565    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ckwn\" (UniqueName: \"kubernetes.io/projected/87f23220-3a1e-457b-9f44-8e64fc4e131e-kube-api-access-4ckwn\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf\" (UID: \"87f23220-3a1e-457b-9f44-8e64fc4e131e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"
Oct 31 12:45:00.558747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.550190    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crwtd\" (UniqueName: \"kubernetes.io/projected/26790f61-8a65-4d3a-8927-c6bb7406b216-kube-api-access-crwtd\") pod \"maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck\" (UID: \"26790f61-8a65-4d3a-8927-c6bb7406b216\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"
Oct 31 12:45:00.575989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.561837    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"
Oct 31 12:45:00.579718 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.579685    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"
Oct 31 12:45:00.595831 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:00.595715    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"
Oct 31 12:45:00.719909 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:00.716995    5663 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7029-08c6ba\" is forbidden: User \"system:node:ip-10-0-140-137.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-137.us-east-2.compute.internal' and this object" podUID="9d0d745e-e42a-47a1-aff3-e8d172dbdfd0" pod="opendatahub/test-curl-7029-08c6ba"
Oct 31 12:45:01.050222 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.048263    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"]
Oct 31 12:45:01.228764 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:45:01.228723    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6d164c_805d_4977_b40a_20cc011edfb3.slice/crio-ab90444b235dc470b0c175fe274842c9d1728da8da109515a0ecee68da6a197a WatchSource:0}: Error finding container ab90444b235dc470b0c175fe274842c9d1728da8da109515a0ecee68da6a197a: Status 404 returned error can't find the container with id ab90444b235dc470b0c175fe274842c9d1728da8da109515a0ecee68da6a197a
Oct 31 12:45:01.241577 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.241533    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"]
Oct 31 12:45:01.267640 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.266236    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"]
Oct 31 12:45:01.269631 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:45:01.269603    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f23220_3a1e_457b_9f44_8e64fc4e131e.slice/crio-567e6729ce54e7f0391a1e66dec05a4370000494847161023517e7d989bb7d97 WatchSource:0}: Error finding container 567e6729ce54e7f0391a1e66dec05a4370000494847161023517e7d989bb7d97: Status 404 returned error can't find the container with id 567e6729ce54e7f0391a1e66dec05a4370000494847161023517e7d989bb7d97
Oct 31 12:45:01.311135 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.307572    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"]
Oct 31 12:45:01.315082 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:45:01.315051    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961f2b49_e1fc_4d28_93f5_880f791be763.slice/crio-667bebbe43fa6373facdfbccc9729d6d8eec0729c9286b069997e6f702b18cf0 WatchSource:0}: Error finding container 667bebbe43fa6373facdfbccc9729d6d8eec0729c9286b069997e6f702b18cf0: Status 404 returned error can't find the container with id 667bebbe43fa6373facdfbccc9729d6d8eec0729c9286b069997e6f702b18cf0
Oct 31 12:45:01.394472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.394412    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"]
Oct 31 12:45:01.476665 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.476633    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0d745e-e42a-47a1-aff3-e8d172dbdfd0" path="/var/lib/kubelet/pods/9d0d745e-e42a-47a1-aff3-e8d172dbdfd0/volumes"
Oct 31 12:45:01.541621 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.541535    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp" event={"ID":"f98924ce-f10b-45fd-8a83-9f8ebfa424f8","Type":"ContainerStarted","Data":"739a48b3e295afc487a202ef0e139f67fc25b37ff59704a6837c1869d8bf373c"}
Oct 31 12:45:01.542858 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.542827    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t" event={"ID":"961f2b49-e1fc-4d28-93f5-880f791be763","Type":"ContainerStarted","Data":"667bebbe43fa6373facdfbccc9729d6d8eec0729c9286b069997e6f702b18cf0"}
Oct 31 12:45:01.544270 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.544238    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf" event={"ID":"87f23220-3a1e-457b-9f44-8e64fc4e131e","Type":"ContainerStarted","Data":"567e6729ce54e7f0391a1e66dec05a4370000494847161023517e7d989bb7d97"}
Oct 31 12:45:01.545623 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.545603    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl" event={"ID":"1c6d164c-805d-4977-b40a-20cc011edfb3","Type":"ContainerStarted","Data":"ab90444b235dc470b0c175fe274842c9d1728da8da109515a0ecee68da6a197a"}
Oct 31 12:45:01.547322 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.547293    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck" event={"ID":"26790f61-8a65-4d3a-8927-c6bb7406b216","Type":"ContainerStarted","Data":"ff8507efb5bdd9ceee98912ede11ae5e1402af48dbd64a0a92c8644644f5d33e"}
Oct 31 12:45:01.547465 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.547415    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7029-87fbbc" podUID="82ea9df7-76a8-4c74-90ac-4d5d5d312644" containerName="test-curl-7029-87fbbc" containerID="cri-o://5d0b9a1f06a60927d92dc3dbfce955a1b822b82a941a9168d14688bcc02f0a1c" gracePeriod=2
Oct 31 12:45:01.577182 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.577152    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"]
Oct 31 12:45:01.578062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.578039    5663 cpu_manager.go:414] "containerMap: removing container" podUID="82ea9df7-76a8-4c74-90ac-4d5d5d312644" containerName="test-curl-7029-87fbbc"
Oct 31 12:45:01.578175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.578064    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ea9df7-76a8-4c74-90ac-4d5d5d312644" containerName="test-curl-7029-87fbbc"
Oct 31 12:45:01.578253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.578235    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ea9df7-76a8-4c74-90ac-4d5d5d312644" containerName="test-curl-7029-87fbbc"
Oct 31 12:45:01.581905 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.581885    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.595571 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.589426    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e1fc0ffe-openshift-default-dockercfg-2nk8m\""
Oct 31 12:45:01.606381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.600096    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"]
Oct 31 12:45:01.770511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.760904    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-credential-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.770511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.760982    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db35ea34-5527-4982-8a90-3bf05ef57357-istiod-ca-cert\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.770511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.761063    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjvf\" (UniqueName: \"kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-kube-api-access-5wjvf\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.770511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.761095    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-certs\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.770511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.761186    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-data\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.770511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.761242    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db35ea34-5527-4982-8a90-3bf05ef57357-istio-podinfo\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.770511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.761276    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-envoy\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.770511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.761303    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.770511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.761477    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-istio-token\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.955700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.944748    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db35ea34-5527-4982-8a90-3bf05ef57357-istio-podinfo\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.955700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.944824    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-envoy\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.955700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.944883    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.955700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.945120    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-istio-token\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.955700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.948515    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-credential-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.955700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.948609    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db35ea34-5527-4982-8a90-3bf05ef57357-istiod-ca-cert\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.955700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.948734    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjvf\" (UniqueName: \"kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-kube-api-access-5wjvf\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.955700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.948768    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-certs\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.955700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.948894    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-data\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.955700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.950748    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-credential-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.997654 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.968002    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.997654 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.969012    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db35ea34-5527-4982-8a90-3bf05ef57357-istiod-ca-cert\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.997654 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.969247    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-certs\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:01.997654 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:01.969486    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-data\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:02.145899 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.145872    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-envoy\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:02.148499 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.148472    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-istio-token\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:02.149012 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.148993    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjvf\" (UniqueName: \"kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-kube-api-access-5wjvf\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:02.149523 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.149505    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db35ea34-5527-4982-8a90-3bf05ef57357-istio-podinfo\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:02.258626 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.258545    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:02.302781 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.302555    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-ad8af6"
Oct 31 12:45:02.380189 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.380158    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4-kube-api-access-bfmf2\" (UniqueName: \"kubernetes.io/projected/2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4-kube-api-access-bfmf2\") pod \"2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4\" (UID: \"2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4\") "
Oct 31 12:45:02.382983 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.382953    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4-kube-api-access-bfmf2" pod "2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4" (UID: "2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4"). InnerVolumeSpecName "kube-api-access-bfmf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:02.484691 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.484648    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfmf2\" (UniqueName: \"kubernetes.io/projected/2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4-kube-api-access-bfmf2\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:02.627748 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.627712    5663 generic.go:356] "Generic (PLEG): container finished" podID="2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4" containerID="cad9d30bc8f6616c340b695ffa08c0dccb156d17438a2046f910264796acd6bf" exitCode=137
Oct 31 12:45:02.628303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.628007    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-ad8af6"
Oct 31 12:45:02.628623 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.628065    5663 scope.go:122] "RemoveContainer" containerID="cad9d30bc8f6616c340b695ffa08c0dccb156d17438a2046f910264796acd6bf"
Oct 31 12:45:02.652409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.652379    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"]
Oct 31 12:45:02.673506 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:02.673465    5663 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7029-ad8af6\" is forbidden: User \"system:node:ip-10-0-140-137.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-137.us-east-2.compute.internal' and this object" podUID="2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4" pod="opendatahub/test-curl-7029-ad8af6"
Oct 31 12:45:02.677037 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:45:02.676952    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb35ea34_5527_4982_8a90_3bf05ef57357.slice/crio-d2f60ac3271fd66ca0ff307e75cee0e47961a3f8888e7e20df25adaf5cb28c0d WatchSource:0}: Error finding container d2f60ac3271fd66ca0ff307e75cee0e47961a3f8888e7e20df25adaf5cb28c0d: Status 404 returned error can't find the container with id d2f60ac3271fd66ca0ff307e75cee0e47961a3f8888e7e20df25adaf5cb28c0d
Oct 31 12:45:02.682396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.681805    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:45:02.682396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.681903    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:45:02.682396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.681950    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:45:02.699037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.698987    5663 scope.go:122] "RemoveContainer" containerID="cad9d30bc8f6616c340b695ffa08c0dccb156d17438a2046f910264796acd6bf"
Oct 31 12:45:02.699717 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:02.699688    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad9d30bc8f6616c340b695ffa08c0dccb156d17438a2046f910264796acd6bf\": container with ID starting with cad9d30bc8f6616c340b695ffa08c0dccb156d17438a2046f910264796acd6bf not found: ID does not exist" containerID="cad9d30bc8f6616c340b695ffa08c0dccb156d17438a2046f910264796acd6bf"
Oct 31 12:45:02.699910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:02.699882    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad9d30bc8f6616c340b695ffa08c0dccb156d17438a2046f910264796acd6bf"} err="failed to get container status \"cad9d30bc8f6616c340b695ffa08c0dccb156d17438a2046f910264796acd6bf\": rpc error: code = NotFound desc = could not find container \"cad9d30bc8f6616c340b695ffa08c0dccb156d17438a2046f910264796acd6bf\": container with ID starting with cad9d30bc8f6616c340b695ffa08c0dccb156d17438a2046f910264796acd6bf not found: ID does not exist"
Oct 31 12:45:03.560480 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:03.560430    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4" path="/var/lib/kubelet/pods/2e25f62f-6bb6-4d4d-8e6e-948b9a54b8f4/volumes"
Oct 31 12:45:03.648759 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:03.648716    5663 generic.go:356] "Generic (PLEG): container finished" podID="82ea9df7-76a8-4c74-90ac-4d5d5d312644" containerID="5d0b9a1f06a60927d92dc3dbfce955a1b822b82a941a9168d14688bcc02f0a1c" exitCode=137
Oct 31 12:45:03.655392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:03.655050    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" event={"ID":"db35ea34-5527-4982-8a90-3bf05ef57357","Type":"ContainerStarted","Data":"42ae02304d162dfbe913c6a283f7d5c9db63673c03682651c5d70d5b6f5bd2b0"}
Oct 31 12:45:03.655392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:03.655099    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" event={"ID":"db35ea34-5527-4982-8a90-3bf05ef57357","Type":"ContainerStarted","Data":"d2f60ac3271fd66ca0ff307e75cee0e47961a3f8888e7e20df25adaf5cb28c0d"}
Oct 31 12:45:03.696222 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:03.696168    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" podStartSLOduration=2.696152969 podStartE2EDuration="2.696152969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:03.695620647 +0000 UTC m=+1028.436046863" watchObservedRunningTime="2026-10-31 12:45:03.696152969 +0000 UTC m=+1028.436579182"
Oct 31 12:45:04.012461 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.012426    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-87fbbc"
Oct 31 12:45:04.082166 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.082130    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ea9df7-76a8-4c74-90ac-4d5d5d312644-kube-api-access-57mss\" (UniqueName: \"kubernetes.io/projected/82ea9df7-76a8-4c74-90ac-4d5d5d312644-kube-api-access-57mss\") pod \"82ea9df7-76a8-4c74-90ac-4d5d5d312644\" (UID: \"82ea9df7-76a8-4c74-90ac-4d5d5d312644\") "
Oct 31 12:45:04.101396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.092439    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ea9df7-76a8-4c74-90ac-4d5d5d312644-kube-api-access-57mss" pod "82ea9df7-76a8-4c74-90ac-4d5d5d312644" (UID: "82ea9df7-76a8-4c74-90ac-4d5d5d312644"). InnerVolumeSpecName "kube-api-access-57mss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:04.195441 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.193932    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57mss\" (UniqueName: \"kubernetes.io/projected/82ea9df7-76a8-4c74-90ac-4d5d5d312644-kube-api-access-57mss\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:04.260367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.260332    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:04.267392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.266876    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"]
Oct 31 12:45:04.280337 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.274972    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:04.280337 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.275069    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"
Oct 31 12:45:04.280337 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.277310    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"]
Oct 31 12:45:04.287340 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.283433    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-8faad6\""
Oct 31 12:45:04.296008 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.295285    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg2kh\" (UniqueName: \"kubernetes.io/projected/8e5921ea-8954-4cf9-be8d-620f35be10e1-kube-api-access-vg2kh\") pod \"payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl\" (UID: \"8e5921ea-8954-4cf9-be8d-620f35be10e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"
Oct 31 12:45:04.296008 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.295606    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5921ea-8954-4cf9-be8d-620f35be10e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl\" (UID: \"8e5921ea-8954-4cf9-be8d-620f35be10e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"
Oct 31 12:45:04.403841 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.399811    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5921ea-8954-4cf9-be8d-620f35be10e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl\" (UID: \"8e5921ea-8954-4cf9-be8d-620f35be10e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"
Oct 31 12:45:04.433006 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.431521    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg2kh\" (UniqueName: \"kubernetes.io/projected/8e5921ea-8954-4cf9-be8d-620f35be10e1-kube-api-access-vg2kh\") pod \"payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl\" (UID: \"8e5921ea-8954-4cf9-be8d-620f35be10e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"
Oct 31 12:45:04.472527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.471258    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5921ea-8954-4cf9-be8d-620f35be10e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl\" (UID: \"8e5921ea-8954-4cf9-be8d-620f35be10e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"
Oct 31 12:45:04.574738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.574670    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg2kh\" (UniqueName: \"kubernetes.io/projected/8e5921ea-8954-4cf9-be8d-620f35be10e1-kube-api-access-vg2kh\") pod \"payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl\" (UID: \"8e5921ea-8954-4cf9-be8d-620f35be10e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"
Oct 31 12:45:04.612162 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.603637    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"
Oct 31 12:45:04.789428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.788250    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-87fbbc"
Oct 31 12:45:04.794693 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.794627    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:04.794693 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.794695    5663 scope.go:122] "RemoveContainer" containerID="5d0b9a1f06a60927d92dc3dbfce955a1b822b82a941a9168d14688bcc02f0a1c"
Oct 31 12:45:04.816916 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:04.816771    5663 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7029-87fbbc\" is forbidden: User \"system:node:ip-10-0-140-137.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-137.us-east-2.compute.internal' and this object" podUID="82ea9df7-76a8-4c74-90ac-4d5d5d312644" pod="opendatahub/test-curl-7029-87fbbc"
Oct 31 12:45:04.818553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:04.818185    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:45:04.829433 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:04.823069    5663 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7029-87fbbc\" is forbidden: User \"system:node:ip-10-0-140-137.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-137.us-east-2.compute.internal' and this object" podUID="82ea9df7-76a8-4c74-90ac-4d5d5d312644" pod="opendatahub/test-curl-7029-87fbbc"
Oct 31 12:45:05.165341 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:05.165285    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"]
Oct 31 12:45:05.167182 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:45:05.167151    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5921ea_8954_4cf9_be8d_620f35be10e1.slice/crio-524e889789e7261422dc4f7dab6a368c4f6d227b7ac024ae3cfb014d59bc7f26 WatchSource:0}: Error finding container 524e889789e7261422dc4f7dab6a368c4f6d227b7ac024ae3cfb014d59bc7f26: Status 404 returned error can't find the container with id 524e889789e7261422dc4f7dab6a368c4f6d227b7ac024ae3cfb014d59bc7f26
Oct 31 12:45:05.462703 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:05.462592    5663 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7029-87fbbc\" is forbidden: User \"system:node:ip-10-0-140-137.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-137.us-east-2.compute.internal' and this object" podUID="82ea9df7-76a8-4c74-90ac-4d5d5d312644" pod="opendatahub/test-curl-7029-87fbbc"
Oct 31 12:45:05.472893 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:05.472437    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ea9df7-76a8-4c74-90ac-4d5d5d312644" path="/var/lib/kubelet/pods/82ea9df7-76a8-4c74-90ac-4d5d5d312644/volumes"
Oct 31 12:45:05.809481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:05.809449    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl" event={"ID":"8e5921ea-8954-4cf9-be8d-620f35be10e1","Type":"ContainerStarted","Data":"f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9"}
Oct 31 12:45:05.809910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:05.809494    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl" event={"ID":"8e5921ea-8954-4cf9-be8d-620f35be10e1","Type":"ContainerStarted","Data":"524e889789e7261422dc4f7dab6a368c4f6d227b7ac024ae3cfb014d59bc7f26"}
Oct 31 12:45:05.809910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:05.809525    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"
Oct 31 12:45:05.836533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:05.835690    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl" podStartSLOduration=1.835672006 podStartE2EDuration="1.835672006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:05.833713949 +0000 UTC m=+1030.574140169" watchObservedRunningTime="2026-10-31 12:45:05.835672006 +0000 UTC m=+1030.576098222"
Oct 31 12:45:07.660630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.660592    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"]
Oct 31 12:45:07.688957 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.687274    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"]
Oct 31 12:45:07.688957 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.687313    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"]
Oct 31 12:45:07.688957 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.687409    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"
Oct 31 12:45:07.694376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.694324    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e1fc0ffe\""
Oct 31 12:45:07.707206 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.703547    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"
Oct 31 12:45:07.717915 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.707696    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e1fc0ffe-dockercfg-bxwd2\""
Oct 31 12:45:07.737071 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.737038    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"]
Oct 31 12:45:07.741632 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.741597    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af38a90f-5827-452c-b245-7e147b5983eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj\" (UID: \"af38a90f-5827-452c-b245-7e147b5983eb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"
Oct 31 12:45:07.741938 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.741913    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs\" (UID: \"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1\") " pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"
Oct 31 12:45:07.742033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.741989    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjds\" (UniqueName: \"kubernetes.io/projected/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-kube-api-access-mpjds\") pod \"payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs\" (UID: \"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1\") " pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"
Oct 31 12:45:07.742106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.742033    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86wk\" (UniqueName: \"kubernetes.io/projected/af38a90f-5827-452c-b245-7e147b5983eb-kube-api-access-b86wk\") pod \"payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj\" (UID: \"af38a90f-5827-452c-b245-7e147b5983eb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"
Oct 31 12:45:07.867375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.862874    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs\" (UID: \"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1\") " pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"
Oct 31 12:45:07.867375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.862971    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjds\" (UniqueName: \"kubernetes.io/projected/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-kube-api-access-mpjds\") pod \"payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs\" (UID: \"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1\") " pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"
Oct 31 12:45:07.867375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.863090    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86wk\" (UniqueName: \"kubernetes.io/projected/af38a90f-5827-452c-b245-7e147b5983eb-kube-api-access-b86wk\") pod \"payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj\" (UID: \"af38a90f-5827-452c-b245-7e147b5983eb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"
Oct 31 12:45:07.867375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.863188    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af38a90f-5827-452c-b245-7e147b5983eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj\" (UID: \"af38a90f-5827-452c-b245-7e147b5983eb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"
Oct 31 12:45:07.867375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.864454    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af38a90f-5827-452c-b245-7e147b5983eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj\" (UID: \"af38a90f-5827-452c-b245-7e147b5983eb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"
Oct 31 12:45:07.867375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.865301    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs\" (UID: \"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1\") " pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"
Oct 31 12:45:07.996228 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.980773    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86wk\" (UniqueName: \"kubernetes.io/projected/af38a90f-5827-452c-b245-7e147b5983eb-kube-api-access-b86wk\") pod \"payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj\" (UID: \"af38a90f-5827-452c-b245-7e147b5983eb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"
Oct 31 12:45:07.996228 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:07.984322    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjds\" (UniqueName: \"kubernetes.io/projected/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-kube-api-access-mpjds\") pod \"payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs\" (UID: \"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1\") " pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"
Oct 31 12:45:08.032576 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:08.025594    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"
Oct 31 12:45:08.041388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:08.039924    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"
Oct 31 12:45:08.844451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:08.828489    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 12:45:08.880650 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:08.877449    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:09.510437 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:45:09.508145    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d56a73_8e70_4d00_8c7f_fb5ae35ed3c1.slice/crio-db9a4184bb852492ea469a2ab615b5979784c95b6eb36e4a1ae5cb785d3ab0f8 WatchSource:0}: Error finding container db9a4184bb852492ea469a2ab615b5979784c95b6eb36e4a1ae5cb785d3ab0f8: Status 404 returned error can't find the container with id db9a4184bb852492ea469a2ab615b5979784c95b6eb36e4a1ae5cb785d3ab0f8
Oct 31 12:45:09.627069 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:09.618591    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"]
Oct 31 12:45:09.627069 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:09.618650    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"]
Oct 31 12:45:10.263416 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:10.263375    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj" event={"ID":"af38a90f-5827-452c-b245-7e147b5983eb","Type":"ContainerStarted","Data":"ec1e82dc1906eeb6f65112e520bd1cb60952c9f883ce448e8068b3d2609faf1b"}
Oct 31 12:45:10.263896 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:10.263458    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"
Oct 31 12:45:10.263896 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:10.263483    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj" event={"ID":"af38a90f-5827-452c-b245-7e147b5983eb","Type":"ContainerStarted","Data":"5a3dac001951b15ca13799acd297e9f6de27d2fe09da8693d38c3702f1c54364"}
Oct 31 12:45:10.269568 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:10.269536    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs" event={"ID":"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1","Type":"ContainerStarted","Data":"c4eae499a7bbbbc0b7c7031790d0ae8511d5027ca69ac989118b000b8c39d53f"}
Oct 31 12:45:10.269796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:10.269775    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs" event={"ID":"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1","Type":"ContainerStarted","Data":"db9a4184bb852492ea469a2ab615b5979784c95b6eb36e4a1ae5cb785d3ab0f8"}
Oct 31 12:45:10.270821 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:10.270798    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"
Oct 31 12:45:10.339382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:10.330267    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj" podStartSLOduration=3.330245225 podStartE2EDuration="3.330245225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:10.289937908 +0000 UTC m=+1035.030364125" watchObservedRunningTime="2026-10-31 12:45:10.330245225 +0000 UTC m=+1035.070671440"
Oct 31 12:45:11.609220 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:11.609172    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:11.610342 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:11.610313    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:11.626993 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:11.626938    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:11.830052 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:11.829087    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:11.847104 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:11.847065    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:11.891426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:11.862656    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:12.051379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:12.037827    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:12.301428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:12.296671    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:12.629644 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:12.629600    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:12.631170 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:12.631145    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:45:12.656400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:12.646933    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:12.656400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:12.648497    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:45:12.696121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:12.694339    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs" podStartSLOduration=5.6943197439999995 podStartE2EDuration="5.694319744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:10.341599539 +0000 UTC m=+1035.082025755" watchObservedRunningTime="2026-10-31 12:45:12.694319744 +0000 UTC m=+1037.434745963"
Oct 31 12:45:12.762613 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:12.762581    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:45:12.850610 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:12.849479    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:45:12.860410 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:12.857726    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:45:12.896261 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:12.896182    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:45:12.948923 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:12.948894    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:45:13.300645 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:13.300535    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-lsjpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:45:13.300935 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:13.300883    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx" podUID="6cacf32b-0e2d-48cc-a0be-c99e26159bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:15.793372 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:15.793323    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck" event={"ID":"26790f61-8a65-4d3a-8927-c6bb7406b216","Type":"ContainerStarted","Data":"909c715d866a77e8ad5a8fd633a1198982a7fd4f5901c48a28cb7b234941eb9b"}
Oct 31 12:45:15.811747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:15.811697    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck" podStartSLOduration=2.7775415519999997 podStartE2EDuration="15.811682073s" totalImagesPullingTime="13.034140521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:15.809826925 +0000 UTC m=+1040.550253141" watchObservedRunningTime="2026-10-31 12:45:15.811682073 +0000 UTC m=+1040.552108289"
Oct 31 12:45:16.853937 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:16.853891    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t" event={"ID":"961f2b49-e1fc-4d28-93f5-880f791be763","Type":"ContainerStarted","Data":"c61db35ccc69f9b98b0772623bb78902e5143d40ce590cd17656e8bade8b671d"}
Oct 31 12:45:16.910381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:16.903389    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf" event={"ID":"87f23220-3a1e-457b-9f44-8e64fc4e131e","Type":"ContainerStarted","Data":"fdee91ec9a93c854af5e7facfd1dd1e1cb9593cd815b85e75b06dd3d3b71c7a2"}
Oct 31 12:45:16.910649 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:16.910595    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t" podStartSLOduration=3.066082033 podStartE2EDuration="16.902536137s" totalImagesPullingTime="13.836454104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:16.900578984 +0000 UTC m=+1041.641005210" watchObservedRunningTime="2026-10-31 12:45:16.902536137 +0000 UTC m=+1041.642962354"
Oct 31 12:45:16.915875 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:16.914737    5663 generic.go:356] "Generic (PLEG): container finished" podID="1c6d164c-805d-4977-b40a-20cc011edfb3" containerID="6049611e70ce0a6f45143b072ebbf31f7852bf2fd5a169c6ff6efa0a11f5f8f9" exitCode=60
Oct 31 12:45:16.915875 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:16.914891    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl" event={"ID":"1c6d164c-805d-4977-b40a-20cc011edfb3","Type":"ContainerDied","Data":"6049611e70ce0a6f45143b072ebbf31f7852bf2fd5a169c6ff6efa0a11f5f8f9"}
Oct 31 12:45:16.915875 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:16.915374    5663 scope.go:122] "RemoveContainer" containerID="6049611e70ce0a6f45143b072ebbf31f7852bf2fd5a169c6ff6efa0a11f5f8f9"
Oct 31 12:45:16.922035 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:16.922010    5663 generic.go:356] "Generic (PLEG): container finished" podID="26790f61-8a65-4d3a-8927-c6bb7406b216" containerID="909c715d866a77e8ad5a8fd633a1198982a7fd4f5901c48a28cb7b234941eb9b" exitCode=60
Oct 31 12:45:16.922973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:16.922563    5663 scope.go:122] "RemoveContainer" containerID="909c715d866a77e8ad5a8fd633a1198982a7fd4f5901c48a28cb7b234941eb9b"
Oct 31 12:45:16.922973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:16.922698    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck" event={"ID":"26790f61-8a65-4d3a-8927-c6bb7406b216","Type":"ContainerDied","Data":"909c715d866a77e8ad5a8fd633a1198982a7fd4f5901c48a28cb7b234941eb9b"}
Oct 31 12:45:16.928023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:16.926848    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf" podStartSLOduration=2.905848407 podStartE2EDuration="16.92683258s" totalImagesPullingTime="14.020984173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:16.926776016 +0000 UTC m=+1041.667202234" watchObservedRunningTime="2026-10-31 12:45:16.92683258 +0000 UTC m=+1041.667258797"
Oct 31 12:45:16.928023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:16.927740    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"
Oct 31 12:45:16.930613 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:16.930588    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp" event={"ID":"f98924ce-f10b-45fd-8a83-9f8ebfa424f8","Type":"ContainerStarted","Data":"0e470b41a2814582e49454fb2f33aa82735840ed70ac769cfbcc4f4e12b8d764"}
Oct 31 12:45:17.002588 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:17.002537    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp" podStartSLOduration=2.914406735 podStartE2EDuration="17.002524004s" totalImagesPullingTime="14.088117269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:17.001103754 +0000 UTC m=+1041.741529989" watchObservedRunningTime="2026-10-31 12:45:17.002524004 +0000 UTC m=+1041.742950250"
Oct 31 12:45:17.946429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:17.945687    5663 generic.go:356] "Generic (PLEG): container finished" podID="87f23220-3a1e-457b-9f44-8e64fc4e131e" containerID="fdee91ec9a93c854af5e7facfd1dd1e1cb9593cd815b85e75b06dd3d3b71c7a2" exitCode=60
Oct 31 12:45:17.946429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:17.945799    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf" event={"ID":"87f23220-3a1e-457b-9f44-8e64fc4e131e","Type":"ContainerDied","Data":"fdee91ec9a93c854af5e7facfd1dd1e1cb9593cd815b85e75b06dd3d3b71c7a2"}
Oct 31 12:45:17.946429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:17.946138    5663 scope.go:122] "RemoveContainer" containerID="fdee91ec9a93c854af5e7facfd1dd1e1cb9593cd815b85e75b06dd3d3b71c7a2"
Oct 31 12:45:17.949017 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:17.948403    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl" event={"ID":"1c6d164c-805d-4977-b40a-20cc011edfb3","Type":"ContainerStarted","Data":"28a0f233e4ace24ad93773e4d40f82294dd6e61f7e39e415f0bd396506376e8a"}
Oct 31 12:45:17.951844 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:17.951751    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck" event={"ID":"26790f61-8a65-4d3a-8927-c6bb7406b216","Type":"ContainerStarted","Data":"fe26090103cc6e352ec9287268942b96fdd522ba999d6effce5e0fe04f02ee66"}
Oct 31 12:45:17.953968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:17.953943    5663 generic.go:356] "Generic (PLEG): container finished" podID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" containerID="0e470b41a2814582e49454fb2f33aa82735840ed70ac769cfbcc4f4e12b8d764" exitCode=60
Oct 31 12:45:17.954083 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:17.954029    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp" event={"ID":"f98924ce-f10b-45fd-8a83-9f8ebfa424f8","Type":"ContainerDied","Data":"0e470b41a2814582e49454fb2f33aa82735840ed70ac769cfbcc4f4e12b8d764"}
Oct 31 12:45:17.954265 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:17.954245    5663 scope.go:122] "RemoveContainer" containerID="0e470b41a2814582e49454fb2f33aa82735840ed70ac769cfbcc4f4e12b8d764"
Oct 31 12:45:17.955772 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:17.955750    5663 generic.go:356] "Generic (PLEG): container finished" podID="961f2b49-e1fc-4d28-93f5-880f791be763" containerID="c61db35ccc69f9b98b0772623bb78902e5143d40ce590cd17656e8bade8b671d" exitCode=60
Oct 31 12:45:17.955863 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:17.955802    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t" event={"ID":"961f2b49-e1fc-4d28-93f5-880f791be763","Type":"ContainerDied","Data":"c61db35ccc69f9b98b0772623bb78902e5143d40ce590cd17656e8bade8b671d"}
Oct 31 12:45:17.956032 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:17.956014    5663 scope.go:122] "RemoveContainer" containerID="c61db35ccc69f9b98b0772623bb78902e5143d40ce590cd17656e8bade8b671d"
Oct 31 12:45:18.054728 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:18.054321    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl" podStartSLOduration=4.2308466750000004 podStartE2EDuration="18.054306517s" totalImagesPullingTime="13.823459842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:18.0507671 +0000 UTC m=+1042.791193317" watchObservedRunningTime="2026-10-31 12:45:18.054306517 +0000 UTC m=+1042.794732729"
Oct 31 12:45:18.967323 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:18.967284    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf" event={"ID":"87f23220-3a1e-457b-9f44-8e64fc4e131e","Type":"ContainerStarted","Data":"e733d56a82545688cccc32333b4ef967e11653e417a568ee8148f2f87abe7e98"}
Oct 31 12:45:18.969253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:18.969223    5663 generic.go:356] "Generic (PLEG): container finished" podID="1c6d164c-805d-4977-b40a-20cc011edfb3" containerID="28a0f233e4ace24ad93773e4d40f82294dd6e61f7e39e415f0bd396506376e8a" exitCode=60
Oct 31 12:45:18.969412 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:18.969307    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl" event={"ID":"1c6d164c-805d-4977-b40a-20cc011edfb3","Type":"ContainerDied","Data":"28a0f233e4ace24ad93773e4d40f82294dd6e61f7e39e415f0bd396506376e8a"}
Oct 31 12:45:18.969412 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:18.969401    5663 scope.go:122] "RemoveContainer" containerID="6049611e70ce0a6f45143b072ebbf31f7852bf2fd5a169c6ff6efa0a11f5f8f9"
Oct 31 12:45:18.969671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:18.969657    5663 scope.go:122] "RemoveContainer" containerID="28a0f233e4ace24ad93773e4d40f82294dd6e61f7e39e415f0bd396506376e8a"
Oct 31 12:45:18.970140 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:18.970116    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl_odh-ai-gateway-infra(1c6d164c-805d-4977-b40a-20cc011edfb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3"
Oct 31 12:45:18.971645 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:18.971623    5663 generic.go:356] "Generic (PLEG): container finished" podID="26790f61-8a65-4d3a-8927-c6bb7406b216" containerID="fe26090103cc6e352ec9287268942b96fdd522ba999d6effce5e0fe04f02ee66" exitCode=60
Oct 31 12:45:18.971738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:18.971708    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck" event={"ID":"26790f61-8a65-4d3a-8927-c6bb7406b216","Type":"ContainerDied","Data":"fe26090103cc6e352ec9287268942b96fdd522ba999d6effce5e0fe04f02ee66"}
Oct 31 12:45:18.971989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:18.971969    5663 scope.go:122] "RemoveContainer" containerID="fe26090103cc6e352ec9287268942b96fdd522ba999d6effce5e0fe04f02ee66"
Oct 31 12:45:18.972403 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:18.972376    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck_odh-ai-gateway-infra(26790f61-8a65-4d3a-8927-c6bb7406b216)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216"
Oct 31 12:45:18.973818 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:18.973797    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp" event={"ID":"f98924ce-f10b-45fd-8a83-9f8ebfa424f8","Type":"ContainerStarted","Data":"3c2d0e0d7567082d4f6b828f688403ea50bdeac0dd4e108d84a38b15a749ece3"}
Oct 31 12:45:18.975674 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:18.975652    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t" event={"ID":"961f2b49-e1fc-4d28-93f5-880f791be763","Type":"ContainerStarted","Data":"b5abaa18d845b105bcd4e83c470205489b9c87891bd348fd8b21b5b865ebfab8"}
Oct 31 12:45:18.982714 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:18.982688    5663 scope.go:122] "RemoveContainer" containerID="909c715d866a77e8ad5a8fd633a1198982a7fd4f5901c48a28cb7b234941eb9b"
Oct 31 12:45:19.985288 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.985252    5663 generic.go:356] "Generic (PLEG): container finished" podID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" containerID="3c2d0e0d7567082d4f6b828f688403ea50bdeac0dd4e108d84a38b15a749ece3" exitCode=60
Oct 31 12:45:19.985799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.985340    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp" event={"ID":"f98924ce-f10b-45fd-8a83-9f8ebfa424f8","Type":"ContainerDied","Data":"3c2d0e0d7567082d4f6b828f688403ea50bdeac0dd4e108d84a38b15a749ece3"}
Oct 31 12:45:19.985799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.985423    5663 scope.go:122] "RemoveContainer" containerID="0e470b41a2814582e49454fb2f33aa82735840ed70ac769cfbcc4f4e12b8d764"
Oct 31 12:45:19.985799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.985774    5663 scope.go:122] "RemoveContainer" containerID="3c2d0e0d7567082d4f6b828f688403ea50bdeac0dd4e108d84a38b15a749ece3"
Oct 31 12:45:19.986249 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:19.986225    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp_odh-ai-gateway-infra(f98924ce-f10b-45fd-8a83-9f8ebfa424f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8"
Oct 31 12:45:19.987611 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.987585    5663 generic.go:356] "Generic (PLEG): container finished" podID="961f2b49-e1fc-4d28-93f5-880f791be763" containerID="b5abaa18d845b105bcd4e83c470205489b9c87891bd348fd8b21b5b865ebfab8" exitCode=60
Oct 31 12:45:19.987741 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.987666    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t" event={"ID":"961f2b49-e1fc-4d28-93f5-880f791be763","Type":"ContainerDied","Data":"b5abaa18d845b105bcd4e83c470205489b9c87891bd348fd8b21b5b865ebfab8"}
Oct 31 12:45:19.987941 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.987926    5663 scope.go:122] "RemoveContainer" containerID="b5abaa18d845b105bcd4e83c470205489b9c87891bd348fd8b21b5b865ebfab8"
Oct 31 12:45:19.988308 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:19.988285    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t_odh-ai-gateway-infra(961f2b49-e1fc-4d28-93f5-880f791be763)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t" podUID="961f2b49-e1fc-4d28-93f5-880f791be763"
Oct 31 12:45:19.989655 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.989629    5663 generic.go:356] "Generic (PLEG): container finished" podID="87f23220-3a1e-457b-9f44-8e64fc4e131e" containerID="e733d56a82545688cccc32333b4ef967e11653e417a568ee8148f2f87abe7e98" exitCode=60
Oct 31 12:45:19.989739 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.989702    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf" event={"ID":"87f23220-3a1e-457b-9f44-8e64fc4e131e","Type":"ContainerDied","Data":"e733d56a82545688cccc32333b4ef967e11653e417a568ee8148f2f87abe7e98"}
Oct 31 12:45:19.989991 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.989973    5663 scope.go:122] "RemoveContainer" containerID="e733d56a82545688cccc32333b4ef967e11653e417a568ee8148f2f87abe7e98"
Oct 31 12:45:19.990327 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:19.990307    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf_odh-ai-gateway-infra(87f23220-3a1e-457b-9f44-8e64fc4e131e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e"
Oct 31 12:45:19.993185 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.993161    5663 scope.go:122] "RemoveContainer" containerID="28a0f233e4ace24ad93773e4d40f82294dd6e61f7e39e415f0bd396506376e8a"
Oct 31 12:45:19.993564 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:19.993538    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl_odh-ai-gateway-infra(1c6d164c-805d-4977-b40a-20cc011edfb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3"
Oct 31 12:45:19.994963 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.994941    5663 scope.go:122] "RemoveContainer" containerID="fe26090103cc6e352ec9287268942b96fdd522ba999d6effce5e0fe04f02ee66"
Oct 31 12:45:19.995269 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:19.995246    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck_odh-ai-gateway-infra(26790f61-8a65-4d3a-8927-c6bb7406b216)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216"
Oct 31 12:45:19.999417 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:19.999399    5663 scope.go:122] "RemoveContainer" containerID="c61db35ccc69f9b98b0772623bb78902e5143d40ce590cd17656e8bade8b671d"
Oct 31 12:45:20.016449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:20.016424    5663 scope.go:122] "RemoveContainer" containerID="fdee91ec9a93c854af5e7facfd1dd1e1cb9593cd815b85e75b06dd3d3b71c7a2"
Oct 31 12:45:21.004531 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:21.004505    5663 scope.go:122] "RemoveContainer" containerID="3c2d0e0d7567082d4f6b828f688403ea50bdeac0dd4e108d84a38b15a749ece3"
Oct 31 12:45:21.005016 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:21.004980    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp_odh-ai-gateway-infra(f98924ce-f10b-45fd-8a83-9f8ebfa424f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8"
Oct 31 12:45:21.006307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:21.006284    5663 scope.go:122] "RemoveContainer" containerID="b5abaa18d845b105bcd4e83c470205489b9c87891bd348fd8b21b5b865ebfab8"
Oct 31 12:45:21.006703 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:21.006681    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t_odh-ai-gateway-infra(961f2b49-e1fc-4d28-93f5-880f791be763)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t" podUID="961f2b49-e1fc-4d28-93f5-880f791be763"
Oct 31 12:45:21.008049 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:21.008026    5663 scope.go:122] "RemoveContainer" containerID="e733d56a82545688cccc32333b4ef967e11653e417a568ee8148f2f87abe7e98"
Oct 31 12:45:21.008329 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:21.008302    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf_odh-ai-gateway-infra(87f23220-3a1e-457b-9f44-8e64fc4e131e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e"
Oct 31 12:45:21.295262 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:21.295174    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"
Oct 31 12:45:21.909534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:21.909499    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 12:45:21.909728 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:21.909570    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 12:45:22.016865 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:22.016835    5663 generic.go:356] "Generic (PLEG): container finished" podID="342f09b7-245d-4235-84da-8625ac04f790" containerID="cbec050c1d07b9c5f7c745d6de64289e0d797b2b0db9731ae7566049fed690ce" exitCode=0
Oct 31 12:45:22.017249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:22.016909    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" event={"ID":"342f09b7-245d-4235-84da-8625ac04f790","Type":"ContainerDied","Data":"cbec050c1d07b9c5f7c745d6de64289e0d797b2b0db9731ae7566049fed690ce"}
Oct 31 12:45:22.017446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:22.017430    5663 scope.go:122] "RemoveContainer" containerID="cbec050c1d07b9c5f7c745d6de64289e0d797b2b0db9731ae7566049fed690ce"
Oct 31 12:45:22.019260 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:22.019232    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:45:22.019335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:22.019301    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:45:22.019401 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:22.019335    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:45:22.490945 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:22.490898    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:45:22.611617 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:22.611569    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"
Oct 31 12:45:23.104990 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:23.104949    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" event={"ID":"342f09b7-245d-4235-84da-8625ac04f790","Type":"ContainerStarted","Data":"94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb"}
Oct 31 12:45:23.813161 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:23.813125    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:45:23.813550 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:23.813526    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 12:45:23.813727 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:23.813702    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 12:45:24.811536 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:24.811493    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 12:45:24.812048 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:24.811556    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 12:45:25.811758 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:25.811723    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 12:45:25.812271 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:25.811779    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 12:45:26.813499 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:26.811966    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 12:45:26.813499 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:26.812083    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 12:45:27.850374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:27.850147    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 12:45:27.850374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:27.850205    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 12:45:28.884506 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:28.884473    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:45:28.886283 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:28.886260    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:45:28.886975 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:28.886957    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:45:30.444380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:30.438344    5663 scope.go:122] "RemoveContainer" containerID="28a0f233e4ace24ad93773e4d40f82294dd6e61f7e39e415f0bd396506376e8a"
Oct 31 12:45:31.213534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:31.213489    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl" event={"ID":"1c6d164c-805d-4977-b40a-20cc011edfb3","Type":"ContainerStarted","Data":"a26246e7ec0e27d9c6456540a5b58f3fab3d8bf9f9b9218fd04007031ff6f8d3"}
Oct 31 12:45:31.466434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:31.465907    5663 scope.go:122] "RemoveContainer" containerID="fe26090103cc6e352ec9287268942b96fdd522ba999d6effce5e0fe04f02ee66"
Oct 31 12:45:32.225194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:32.225104    5663 generic.go:356] "Generic (PLEG): container finished" podID="1c6d164c-805d-4977-b40a-20cc011edfb3" containerID="a26246e7ec0e27d9c6456540a5b58f3fab3d8bf9f9b9218fd04007031ff6f8d3" exitCode=60
Oct 31 12:45:32.225194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:32.225177    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl" event={"ID":"1c6d164c-805d-4977-b40a-20cc011edfb3","Type":"ContainerDied","Data":"a26246e7ec0e27d9c6456540a5b58f3fab3d8bf9f9b9218fd04007031ff6f8d3"}
Oct 31 12:45:32.225450 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:32.225239    5663 scope.go:122] "RemoveContainer" containerID="28a0f233e4ace24ad93773e4d40f82294dd6e61f7e39e415f0bd396506376e8a"
Oct 31 12:45:32.225707 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:32.225689    5663 scope.go:122] "RemoveContainer" containerID="a26246e7ec0e27d9c6456540a5b58f3fab3d8bf9f9b9218fd04007031ff6f8d3"
Oct 31 12:45:32.226162 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:32.226137    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl_odh-ai-gateway-infra(1c6d164c-805d-4977-b40a-20cc011edfb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3"
Oct 31 12:45:32.227536 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:32.227510    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck" event={"ID":"26790f61-8a65-4d3a-8927-c6bb7406b216","Type":"ContainerStarted","Data":"d464f8cd448ba6f4519bb8b3fd03ec5fc4fb7481a722fa6509d67cd49aadd4d0"}
Oct 31 12:45:32.262376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:32.259571    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"]
Oct 31 12:45:32.438400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:32.435128    5663 scope.go:122] "RemoveContainer" containerID="3c2d0e0d7567082d4f6b828f688403ea50bdeac0dd4e108d84a38b15a749ece3"
Oct 31 12:45:32.438400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:32.436520    5663 scope.go:122] "RemoveContainer" containerID="b5abaa18d845b105bcd4e83c470205489b9c87891bd348fd8b21b5b865ebfab8"
Oct 31 12:45:33.239626 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.239590    5663 generic.go:356] "Generic (PLEG): container finished" podID="26790f61-8a65-4d3a-8927-c6bb7406b216" containerID="d464f8cd448ba6f4519bb8b3fd03ec5fc4fb7481a722fa6509d67cd49aadd4d0" exitCode=60
Oct 31 12:45:33.240055 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.239657    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck" event={"ID":"26790f61-8a65-4d3a-8927-c6bb7406b216","Type":"ContainerDied","Data":"d464f8cd448ba6f4519bb8b3fd03ec5fc4fb7481a722fa6509d67cd49aadd4d0"}
Oct 31 12:45:33.240055 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.239728    5663 scope.go:122] "RemoveContainer" containerID="fe26090103cc6e352ec9287268942b96fdd522ba999d6effce5e0fe04f02ee66"
Oct 31 12:45:33.240207 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.240187    5663 scope.go:122] "RemoveContainer" containerID="d464f8cd448ba6f4519bb8b3fd03ec5fc4fb7481a722fa6509d67cd49aadd4d0"
Oct 31 12:45:33.240660 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:33.240634    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck_odh-ai-gateway-infra(26790f61-8a65-4d3a-8927-c6bb7406b216)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216"
Oct 31 12:45:33.244002 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.243977    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp" event={"ID":"f98924ce-f10b-45fd-8a83-9f8ebfa424f8","Type":"ContainerStarted","Data":"a9e09a23669fd94a5d42019f0590d0996f23a2a0aebd25baaa56e13ca8f3382b"}
Oct 31 12:45:33.246005 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.245979    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t" event={"ID":"961f2b49-e1fc-4d28-93f5-880f791be763","Type":"ContainerStarted","Data":"49d2d7b43c0324ab4854043860f3466196733189a418da29d6e2bd27b1d249cb"}
Oct 31 12:45:33.301049 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.301012    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"]
Oct 31 12:45:33.432655 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.432624    5663 scope.go:122] "RemoveContainer" containerID="e733d56a82545688cccc32333b4ef967e11653e417a568ee8148f2f87abe7e98"
Oct 31 12:45:33.444593 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.444565    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"
Oct 31 12:45:33.545583 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.545548    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6d164c-805d-4977-b40a-20cc011edfb3-kube-api-access-5rkd7\" (UniqueName: \"kubernetes.io/projected/1c6d164c-805d-4977-b40a-20cc011edfb3-kube-api-access-5rkd7\") pod \"1c6d164c-805d-4977-b40a-20cc011edfb3\" (UID: \"1c6d164c-805d-4977-b40a-20cc011edfb3\") "
Oct 31 12:45:33.545745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.545679    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6d164c-805d-4977-b40a-20cc011edfb3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6d164c-805d-4977-b40a-20cc011edfb3-maas-api-service-ca\") pod \"1c6d164c-805d-4977-b40a-20cc011edfb3\" (UID: \"1c6d164c-805d-4977-b40a-20cc011edfb3\") "
Oct 31 12:45:33.546207 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.546089    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6d164c-805d-4977-b40a-20cc011edfb3-maas-api-service-ca" pod "1c6d164c-805d-4977-b40a-20cc011edfb3" (UID: "1c6d164c-805d-4977-b40a-20cc011edfb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:33.547977 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.547184    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6d164c-805d-4977-b40a-20cc011edfb3-maas-api-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:33.553559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.553532    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6d164c-805d-4977-b40a-20cc011edfb3-kube-api-access-5rkd7" pod "1c6d164c-805d-4977-b40a-20cc011edfb3" (UID: "1c6d164c-805d-4977-b40a-20cc011edfb3"). InnerVolumeSpecName "kube-api-access-5rkd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:33.649276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.649238    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rkd7\" (UniqueName: \"kubernetes.io/projected/1c6d164c-805d-4977-b40a-20cc011edfb3-kube-api-access-5rkd7\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:33.733376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.733328    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"]
Oct 31 12:45:33.734183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.734161    5663 cpu_manager.go:414] "containerMap: removing container" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3" containerName="cleanup"
Oct 31 12:45:33.734183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.734182    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3" containerName="cleanup"
Oct 31 12:45:33.734332 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.734199    5663 cpu_manager.go:414] "containerMap: removing container" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3" containerName="cleanup"
Oct 31 12:45:33.734332 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.734204    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3" containerName="cleanup"
Oct 31 12:45:33.734420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.734336    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3" containerName="cleanup"
Oct 31 12:45:33.734420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.734348    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3" containerName="cleanup"
Oct 31 12:45:33.734420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.734373    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3" containerName="cleanup"
Oct 31 12:45:33.742655 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.742629    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.756217 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.756186    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e1fc0ffe\"/\"openshift-service-ca.crt\""
Oct 31 12:45:33.756475 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.756452    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"]
Oct 31 12:45:33.757098 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.757056    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e1fc0ffe\"/\"e2e-lifecycle-model-e1fc0ffe-kserve-self-signed-certs\""
Oct 31 12:45:33.757206 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.757192    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e1fc0ffe\"/\"default-dockercfg-9mb44\""
Oct 31 12:45:33.757276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.757267    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e1fc0ffe\"/\"kube-root-ca.crt\""
Oct 31 12:45:33.861702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.860577    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-home\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.861702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.860680    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6g9f\" (UniqueName: \"kubernetes.io/projected/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-kube-api-access-q6g9f\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.861702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.860744    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-dshm\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.861702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.860775    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-model-cache\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.861702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.860842    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tmp-dir\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.861702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.861130    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tls-certs\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.971484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.965519    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-home\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.971484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.965630    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6g9f\" (UniqueName: \"kubernetes.io/projected/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-kube-api-access-q6g9f\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.971484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.965695    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-dshm\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.971484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.965726    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-model-cache\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.971484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.965796    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tmp-dir\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.971484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.966054    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tls-certs\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.987960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.974451    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-model-cache\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.987960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.974777    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-home\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:33.987960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:33.975050    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tmp-dir\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:34.008665 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.008609    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-dshm\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:34.026137 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.026107    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tls-certs\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:34.065375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.064187    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6g9f\" (UniqueName: \"kubernetes.io/projected/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-kube-api-access-q6g9f\") pod \"e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") " pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:34.154395 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.153754    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:34.283402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.278094    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp" event={"ID":"f98924ce-f10b-45fd-8a83-9f8ebfa424f8","Type":"ContainerDied","Data":"a9e09a23669fd94a5d42019f0590d0996f23a2a0aebd25baaa56e13ca8f3382b"}
Oct 31 12:45:34.283402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.278210    5663 scope.go:122] "RemoveContainer" containerID="3c2d0e0d7567082d4f6b828f688403ea50bdeac0dd4e108d84a38b15a749ece3"
Oct 31 12:45:34.283402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.278679    5663 scope.go:122] "RemoveContainer" containerID="a9e09a23669fd94a5d42019f0590d0996f23a2a0aebd25baaa56e13ca8f3382b"
Oct 31 12:45:34.283402 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:34.279090    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp_odh-ai-gateway-infra(f98924ce-f10b-45fd-8a83-9f8ebfa424f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8"
Oct 31 12:45:34.283402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.281196    5663 generic.go:356] "Generic (PLEG): container finished" podID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" containerID="a9e09a23669fd94a5d42019f0590d0996f23a2a0aebd25baaa56e13ca8f3382b" exitCode=60
Oct 31 12:45:34.287375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.286112    5663 generic.go:356] "Generic (PLEG): container finished" podID="961f2b49-e1fc-4d28-93f5-880f791be763" containerID="49d2d7b43c0324ab4854043860f3466196733189a418da29d6e2bd27b1d249cb" exitCode=60
Oct 31 12:45:34.287375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.286453    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t" event={"ID":"961f2b49-e1fc-4d28-93f5-880f791be763","Type":"ContainerDied","Data":"49d2d7b43c0324ab4854043860f3466196733189a418da29d6e2bd27b1d249cb"}
Oct 31 12:45:34.287375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.286958    5663 scope.go:122] "RemoveContainer" containerID="49d2d7b43c0324ab4854043860f3466196733189a418da29d6e2bd27b1d249cb"
Oct 31 12:45:34.287618 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:34.287382    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t_odh-ai-gateway-infra(961f2b49-e1fc-4d28-93f5-880f791be763)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t" podUID="961f2b49-e1fc-4d28-93f5-880f791be763"
Oct 31 12:45:34.293374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.290973    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf" event={"ID":"87f23220-3a1e-457b-9f44-8e64fc4e131e","Type":"ContainerStarted","Data":"9394a89d3e8bb48ab299f286ec144e4c22686bd2404e1814929d05226fc61604"}
Oct 31 12:45:34.356375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.351223    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl" event={"ID":"1c6d164c-805d-4977-b40a-20cc011edfb3","Type":"ContainerDied","Data":"ab90444b235dc470b0c175fe274842c9d1728da8da109515a0ecee68da6a197a"}
Oct 31 12:45:34.356375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.351412    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"
Oct 31 12:45:34.356375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.355485    5663 scope.go:122] "RemoveContainer" containerID="b5abaa18d845b105bcd4e83c470205489b9c87891bd348fd8b21b5b865ebfab8"
Oct 31 12:45:34.369186 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.366615    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"]
Oct 31 12:45:34.395379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.395066    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"]
Oct 31 12:45:34.438034 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.438002    5663 scope.go:122] "RemoveContainer" containerID="a26246e7ec0e27d9c6456540a5b58f3fab3d8bf9f9b9218fd04007031ff6f8d3"
Oct 31 12:45:34.472452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.472421    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"]
Oct 31 12:45:34.473821 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.473797    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c67de2-29890845-lvrfl"]
Oct 31 12:45:34.720108 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.720071    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"
Oct 31 12:45:34.751176 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.751094    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"]
Oct 31 12:45:34.814242 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.814211    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26790f61-8a65-4d3a-8927-c6bb7406b216-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26790f61-8a65-4d3a-8927-c6bb7406b216-maas-api-service-ca\") pod \"26790f61-8a65-4d3a-8927-c6bb7406b216\" (UID: \"26790f61-8a65-4d3a-8927-c6bb7406b216\") "
Oct 31 12:45:34.814447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.814424    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26790f61-8a65-4d3a-8927-c6bb7406b216-kube-api-access-crwtd\" (UniqueName: \"kubernetes.io/projected/26790f61-8a65-4d3a-8927-c6bb7406b216-kube-api-access-crwtd\") pod \"26790f61-8a65-4d3a-8927-c6bb7406b216\" (UID: \"26790f61-8a65-4d3a-8927-c6bb7406b216\") "
Oct 31 12:45:34.817894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.817850    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26790f61-8a65-4d3a-8927-c6bb7406b216-kube-api-access-crwtd" pod "26790f61-8a65-4d3a-8927-c6bb7406b216" (UID: "26790f61-8a65-4d3a-8927-c6bb7406b216"). InnerVolumeSpecName "kube-api-access-crwtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:34.819655 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.819626    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26790f61-8a65-4d3a-8927-c6bb7406b216-maas-api-service-ca" pod "26790f61-8a65-4d3a-8927-c6bb7406b216" (UID: "26790f61-8a65-4d3a-8927-c6bb7406b216"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:34.929375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.918949    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26790f61-8a65-4d3a-8927-c6bb7406b216-maas-api-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:34.929375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:34.918993    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crwtd\" (UniqueName: \"kubernetes.io/projected/26790f61-8a65-4d3a-8927-c6bb7406b216-kube-api-access-crwtd\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:35.367800 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.367766    5663 generic.go:356] "Generic (PLEG): container finished" podID="87f23220-3a1e-457b-9f44-8e64fc4e131e" containerID="9394a89d3e8bb48ab299f286ec144e4c22686bd2404e1814929d05226fc61604" exitCode=60
Oct 31 12:45:35.368430 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.367844    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf" event={"ID":"87f23220-3a1e-457b-9f44-8e64fc4e131e","Type":"ContainerDied","Data":"9394a89d3e8bb48ab299f286ec144e4c22686bd2404e1814929d05226fc61604"}
Oct 31 12:45:35.368430 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.367919    5663 scope.go:122] "RemoveContainer" containerID="e733d56a82545688cccc32333b4ef967e11653e417a568ee8148f2f87abe7e98"
Oct 31 12:45:35.368555 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.368431    5663 scope.go:122] "RemoveContainer" containerID="9394a89d3e8bb48ab299f286ec144e4c22686bd2404e1814929d05226fc61604"
Oct 31 12:45:35.368857 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:35.368827    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf_odh-ai-gateway-infra(87f23220-3a1e-457b-9f44-8e64fc4e131e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e"
Oct 31 12:45:35.370769 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.370746    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w" event={"ID":"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e","Type":"ContainerStarted","Data":"0369126dc99163ed08427e6f7c0189c1e2011c3362afdcf61d66353773e30196"}
Oct 31 12:45:35.370876 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.370782    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w" event={"ID":"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e","Type":"ContainerStarted","Data":"1f7eeaea1f6de90546c80efc6eed01ce0656bbf19f2530756acd3c5ab946ee0d"}
Oct 31 12:45:35.372231 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.372213    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"
Oct 31 12:45:35.372315 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.372256    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck" event={"ID":"26790f61-8a65-4d3a-8927-c6bb7406b216","Type":"ContainerDied","Data":"ff8507efb5bdd9ceee98912ede11ae5e1402af48dbd64a0a92c8644644f5d33e"}
Oct 31 12:45:35.384052 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.384035    5663 scope.go:122] "RemoveContainer" containerID="d464f8cd448ba6f4519bb8b3fd03ec5fc4fb7481a722fa6509d67cd49aadd4d0"
Oct 31 12:45:35.420374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.419344    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"]
Oct 31 12:45:35.445416 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.444829    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3" path="/var/lib/kubelet/pods/1c6d164c-805d-4977-b40a-20cc011edfb3/volumes"
Oct 31 12:45:35.473908 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.473871    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"]
Oct 31 12:45:35.484835 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.484809    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b218d3-29890845-mnpck"]
Oct 31 12:45:35.526460 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.526399    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w" podStartSLOduration=2.526330924 podStartE2EDuration="2.526330924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:45:35.503882105 +0000 UTC m=+1060.244308333" watchObservedRunningTime="2026-10-31 12:45:35.526330924 +0000 UTC m=+1060.266757142"
Oct 31 12:45:35.585054 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.585013    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"
Oct 31 12:45:35.594737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.594713    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"
Oct 31 12:45:35.911228 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.911191    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961f2b49-e1fc-4d28-93f5-880f791be763-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961f2b49-e1fc-4d28-93f5-880f791be763-maas-api-service-ca\") pod \"961f2b49-e1fc-4d28-93f5-880f791be763\" (UID: \"961f2b49-e1fc-4d28-93f5-880f791be763\") "
Oct 31 12:45:35.911457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.911308    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-maas-api-service-ca\") pod \"f98924ce-f10b-45fd-8a83-9f8ebfa424f8\" (UID: \"f98924ce-f10b-45fd-8a83-9f8ebfa424f8\") "
Oct 31 12:45:35.911457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.911444    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961f2b49-e1fc-4d28-93f5-880f791be763-kube-api-access-vx2g6\" (UniqueName: \"kubernetes.io/projected/961f2b49-e1fc-4d28-93f5-880f791be763-kube-api-access-vx2g6\") pod \"961f2b49-e1fc-4d28-93f5-880f791be763\" (UID: \"961f2b49-e1fc-4d28-93f5-880f791be763\") "
Oct 31 12:45:35.911592 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.911538    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-kube-api-access-cfwtr\" (UniqueName: \"kubernetes.io/projected/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-kube-api-access-cfwtr\") pod \"f98924ce-f10b-45fd-8a83-9f8ebfa424f8\" (UID: \"f98924ce-f10b-45fd-8a83-9f8ebfa424f8\") "
Oct 31 12:45:35.911671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.911643    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961f2b49-e1fc-4d28-93f5-880f791be763-maas-api-service-ca" pod "961f2b49-e1fc-4d28-93f5-880f791be763" (UID: "961f2b49-e1fc-4d28-93f5-880f791be763"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:35.911727 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.911665    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-maas-api-service-ca" pod "f98924ce-f10b-45fd-8a83-9f8ebfa424f8" (UID: "f98924ce-f10b-45fd-8a83-9f8ebfa424f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:35.912497 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.912480    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961f2b49-e1fc-4d28-93f5-880f791be763-maas-api-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:35.912578 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.912499    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-maas-api-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:35.913603 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.913575    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961f2b49-e1fc-4d28-93f5-880f791be763-kube-api-access-vx2g6" pod "961f2b49-e1fc-4d28-93f5-880f791be763" (UID: "961f2b49-e1fc-4d28-93f5-880f791be763"). InnerVolumeSpecName "kube-api-access-vx2g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:35.913720 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:35.913658    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-kube-api-access-cfwtr" pod "f98924ce-f10b-45fd-8a83-9f8ebfa424f8" (UID: "f98924ce-f10b-45fd-8a83-9f8ebfa424f8"). InnerVolumeSpecName "kube-api-access-cfwtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:36.014190 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.014106    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx2g6\" (UniqueName: \"kubernetes.io/projected/961f2b49-e1fc-4d28-93f5-880f791be763-kube-api-access-vx2g6\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:36.014190 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.014143    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwtr\" (UniqueName: \"kubernetes.io/projected/f98924ce-f10b-45fd-8a83-9f8ebfa424f8-kube-api-access-cfwtr\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:36.106149 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.106114    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"]
Oct 31 12:45:36.163378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.156028    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:36.467376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.459203    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp" event={"ID":"f98924ce-f10b-45fd-8a83-9f8ebfa424f8","Type":"ContainerDied","Data":"739a48b3e295afc487a202ef0e139f67fc25b37ff59704a6837c1869d8bf373c"}
Oct 31 12:45:36.467376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.459283    5663 scope.go:122] "RemoveContainer" containerID="a9e09a23669fd94a5d42019f0590d0996f23a2a0aebd25baaa56e13ca8f3382b"
Oct 31 12:45:36.467376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.459462    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"
Oct 31 12:45:36.494897 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.494871    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"
Oct 31 12:45:36.513203 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.513162    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t" event={"ID":"961f2b49-e1fc-4d28-93f5-880f791be763","Type":"ContainerDied","Data":"667bebbe43fa6373facdfbccc9729d6d8eec0729c9286b069997e6f702b18cf0"}
Oct 31 12:45:36.583954 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.583926    5663 scope.go:122] "RemoveContainer" containerID="49d2d7b43c0324ab4854043860f3466196733189a418da29d6e2bd27b1d249cb"
Oct 31 12:45:36.687529 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.674024    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"]
Oct 31 12:45:36.748452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.742549    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b25d3b-29890845-gptkp"]
Oct 31 12:45:36.761478 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.761433    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"]
Oct 31 12:45:36.767621 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:36.767566    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85ce08-29890845-v6p4t"]
Oct 31 12:45:37.052472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.052442    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"
Oct 31 12:45:37.185409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.175997    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87f23220-3a1e-457b-9f44-8e64fc4e131e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f23220-3a1e-457b-9f44-8e64fc4e131e-maas-api-service-ca\") pod \"87f23220-3a1e-457b-9f44-8e64fc4e131e\" (UID: \"87f23220-3a1e-457b-9f44-8e64fc4e131e\") "
Oct 31 12:45:37.185409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.176148    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f23220-3a1e-457b-9f44-8e64fc4e131e-kube-api-access-4ckwn\" (UniqueName: \"kubernetes.io/projected/87f23220-3a1e-457b-9f44-8e64fc4e131e-kube-api-access-4ckwn\") pod \"87f23220-3a1e-457b-9f44-8e64fc4e131e\" (UID: \"87f23220-3a1e-457b-9f44-8e64fc4e131e\") "
Oct 31 12:45:37.185409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.185266    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f23220-3a1e-457b-9f44-8e64fc4e131e-maas-api-service-ca" pod "87f23220-3a1e-457b-9f44-8e64fc4e131e" (UID: "87f23220-3a1e-457b-9f44-8e64fc4e131e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:37.186756 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.186721    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f23220-3a1e-457b-9f44-8e64fc4e131e-kube-api-access-4ckwn" pod "87f23220-3a1e-457b-9f44-8e64fc4e131e" (UID: "87f23220-3a1e-457b-9f44-8e64fc4e131e"). InnerVolumeSpecName "kube-api-access-4ckwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:37.278992 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.278897    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f23220-3a1e-457b-9f44-8e64fc4e131e-maas-api-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:37.278992 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.278936    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ckwn\" (UniqueName: \"kubernetes.io/projected/87f23220-3a1e-457b-9f44-8e64fc4e131e-kube-api-access-4ckwn\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:37.442137 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.442095    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216" path="/var/lib/kubelet/pods/26790f61-8a65-4d3a-8927-c6bb7406b216/volumes"
Oct 31 12:45:37.442727 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.442704    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961f2b49-e1fc-4d28-93f5-880f791be763" path="/var/lib/kubelet/pods/961f2b49-e1fc-4d28-93f5-880f791be763/volumes"
Oct 31 12:45:37.443247 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.443227    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" path="/var/lib/kubelet/pods/f98924ce-f10b-45fd-8a83-9f8ebfa424f8/volumes"
Oct 31 12:45:37.524540 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.524510    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"
Oct 31 12:45:37.525003 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.524544    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf" event={"ID":"87f23220-3a1e-457b-9f44-8e64fc4e131e","Type":"ContainerDied","Data":"567e6729ce54e7f0391a1e66dec05a4370000494847161023517e7d989bb7d97"}
Oct 31 12:45:37.525003 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.524605    5663 scope.go:122] "RemoveContainer" containerID="9394a89d3e8bb48ab299f286ec144e4c22686bd2404e1814929d05226fc61604"
Oct 31 12:45:37.545483 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.545453    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"]
Oct 31 12:45:37.550801 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:37.550774    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-9b17b9-29890845-pcsjf"]
Oct 31 12:45:39.510380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:39.505065    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e" path="/var/lib/kubelet/pods/87f23220-3a1e-457b-9f44-8e64fc4e131e/volumes"
Oct 31 12:45:41.106781 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.106714    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8" podUID="984090e7-bd56-48d7-933a-a610666c747e" containerName="payload-processing" containerID="cri-o://98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35" gracePeriod=25
Oct 31 12:45:41.384500 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.384475    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"
Oct 31 12:45:41.562207 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.562175    5663 generic.go:356] "Generic (PLEG): container finished" podID="984090e7-bd56-48d7-933a-a610666c747e" containerID="98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35" exitCode=0
Oct 31 12:45:41.562395 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.562217    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"
Oct 31 12:45:41.562395 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.562252    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8" event={"ID":"984090e7-bd56-48d7-933a-a610666c747e","Type":"ContainerDied","Data":"98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35"}
Oct 31 12:45:41.562395 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.562305    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8" event={"ID":"984090e7-bd56-48d7-933a-a610666c747e","Type":"ContainerDied","Data":"a40ecf0d6cc17f5a91d5ff0aec521051a1204c0f74a76b0b18624badbaf8a2ff"}
Oct 31 12:45:41.562395 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.562327    5663 scope.go:122] "RemoveContainer" containerID="98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35"
Oct 31 12:45:41.569285 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.569263    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984090e7-bd56-48d7-933a-a610666c747e-kube-api-access-c82cb\" (UniqueName: \"kubernetes.io/projected/984090e7-bd56-48d7-933a-a610666c747e-kube-api-access-c82cb\") pod \"984090e7-bd56-48d7-933a-a610666c747e\" (UID: \"984090e7-bd56-48d7-933a-a610666c747e\") "
Oct 31 12:45:41.569438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.569326    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984090e7-bd56-48d7-933a-a610666c747e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984090e7-bd56-48d7-933a-a610666c747e-plugins-config-volume\") pod \"984090e7-bd56-48d7-933a-a610666c747e\" (UID: \"984090e7-bd56-48d7-933a-a610666c747e\") "
Oct 31 12:45:41.569693 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.569652    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984090e7-bd56-48d7-933a-a610666c747e-plugins-config-volume" pod "984090e7-bd56-48d7-933a-a610666c747e" (UID: "984090e7-bd56-48d7-933a-a610666c747e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:41.570138 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.570105    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984090e7-bd56-48d7-933a-a610666c747e-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:41.571779 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.571757    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984090e7-bd56-48d7-933a-a610666c747e-kube-api-access-c82cb" pod "984090e7-bd56-48d7-933a-a610666c747e" (UID: "984090e7-bd56-48d7-933a-a610666c747e"). InnerVolumeSpecName "kube-api-access-c82cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:41.573261 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.573244    5663 scope.go:122] "RemoveContainer" containerID="98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35"
Oct 31 12:45:41.573550 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:41.573527    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35\": container with ID starting with 98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35 not found: ID does not exist" containerID="98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35"
Oct 31 12:45:41.573619 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.573560    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35"} err="failed to get container status \"98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35\": rpc error: code = NotFound desc = could not find container \"98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35\": container with ID starting with 98a61bd645e570fc94535816c5c2a1adef63711a55dd22adb993279700e57d35 not found: ID does not exist"
Oct 31 12:45:41.675427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.672330    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c82cb\" (UniqueName: \"kubernetes.io/projected/984090e7-bd56-48d7-933a-a610666c747e-kube-api-access-c82cb\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:41.888682 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.888290    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"]
Oct 31 12:45:41.891804 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:41.891778    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ed25a690-89bf65f7f-jxhh8"]
Oct 31 12:45:43.494104 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:43.494066    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984090e7-bd56-48d7-933a-a610666c747e" path="/var/lib/kubelet/pods/984090e7-bd56-48d7-933a-a610666c747e/volumes"
Oct 31 12:45:44.154250 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:44.154216    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:44.168186 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:44.168158    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:44.672762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:44.672733    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:45:51.927379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:51.916169    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:45:52.017297 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:52.017255    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:52.096572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:52.096528    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:52.110620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:52.098804    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:52.221099 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:52.221005    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"]
Oct 31 12:45:52.280503 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:52.280450    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:45:52.292307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:52.281473    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:52.300516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:52.284790    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerName="istio-proxy" containerID="cri-o://1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46" gracePeriod=30
Oct 31 12:45:52.663764 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:52.658685    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:45:52.702563 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:52.702493    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:52.921434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:52.919464    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:52.921434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:52.920214    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:45:52.977446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:52.975434    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:45:52.998542 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:52.996714    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:53.049408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.019465    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:53.049408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.021127    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:45:53.182679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.149512    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:53.182679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.149934    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:45:53.182679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.152264    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:45:53.182679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.152330    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:45:53.182679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.153375    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:53.530441 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.506459    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:53.712495 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.712454    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:53.738734 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.728725    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:53.739556 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.735147    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:45:53.739747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.739733    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:45:53.739878 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.739868    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:45:53.804116 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.804009    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:53.887783 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:53.879455    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.170402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.168348    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.170402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.168821    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.170402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.169441    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:45:54.189096 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.189056    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:45:54.208962 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.208930    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:45:54.232655 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.228437    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.267738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.257269    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:45:54.267738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.258638    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-ms7fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:45:54.267738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.258756    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.267738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.261500    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.294374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.289960    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.319414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.296344    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:45:54.319414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.297262    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.323558 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.323526    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.596437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.556345    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.692377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.692014    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:45:54.777727 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.704823    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 31 12:45:54.830391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.778009    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:54.850438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.778068    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:45:54.864826 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.860614    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:45:54.908930 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:54.900565    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:45:55.221118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.203509    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Oct 31 12:45:55.221118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.203762    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:45:55.267215 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.267174    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:55.270537 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.270495    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:45:55.295074 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.290301    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:55.307991 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.290483    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:55.329704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.329659    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:45:55.357840 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.353792    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:45:55.628070 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.628031    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:45:55.708709 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.708678    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:45:55.709789 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.709133    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:45:55.709990 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.709977    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:45:55.710110 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.710100    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:45:55.710239 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.710225    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:45:55.710401 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.710387    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:45:55.712559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.712526    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:45:55.715112 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:55.715028    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:45:57.689209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.689184    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:45:57.748715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.748683    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-podinfo\") pod \"e193d74c-a488-4a7b-83d2-866bb9edc44f\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") "
Oct 31 12:45:57.748910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.748743    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-token\" (UniqueName: \"kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-token\") pod \"e193d74c-a488-4a7b-83d2-866bb9edc44f\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") "
Oct 31 12:45:57.748910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.748836    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-data\") pod \"e193d74c-a488-4a7b-83d2-866bb9edc44f\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") "
Oct 31 12:45:57.749033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.748958    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-kube-api-access-7zxbl\" (UniqueName: \"kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-kube-api-access-7zxbl\") pod \"e193d74c-a488-4a7b-83d2-866bb9edc44f\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") "
Oct 31 12:45:57.749033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.748989    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e193d74c-a488-4a7b-83d2-866bb9edc44f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e193d74c-a488-4a7b-83d2-866bb9edc44f-istiod-ca-cert\") pod \"e193d74c-a488-4a7b-83d2-866bb9edc44f\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") "
Oct 31 12:45:57.749183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.749165    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-socket\") pod \"e193d74c-a488-4a7b-83d2-866bb9edc44f\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") "
Oct 31 12:45:57.749257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.749204    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-certs\") pod \"e193d74c-a488-4a7b-83d2-866bb9edc44f\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") "
Oct 31 12:45:57.749257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.749250    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-envoy\") pod \"e193d74c-a488-4a7b-83d2-866bb9edc44f\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") "
Oct 31 12:45:57.749391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.749308    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-credential-socket\") pod \"e193d74c-a488-4a7b-83d2-866bb9edc44f\" (UID: \"e193d74c-a488-4a7b-83d2-866bb9edc44f\") "
Oct 31 12:45:57.749576 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.749551    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-data" pod "e193d74c-a488-4a7b-83d2-866bb9edc44f" (UID: "e193d74c-a488-4a7b-83d2-866bb9edc44f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:45:57.749787 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.749764    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-socket" pod "e193d74c-a488-4a7b-83d2-866bb9edc44f" (UID: "e193d74c-a488-4a7b-83d2-866bb9edc44f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:45:57.750484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.750086    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-certs" pod "e193d74c-a488-4a7b-83d2-866bb9edc44f" (UID: "e193d74c-a488-4a7b-83d2-866bb9edc44f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:45:57.750484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.750126    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e193d74c-a488-4a7b-83d2-866bb9edc44f-istiod-ca-cert" pod "e193d74c-a488-4a7b-83d2-866bb9edc44f" (UID: "e193d74c-a488-4a7b-83d2-866bb9edc44f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:45:57.750484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.750330    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-credential-socket" pod "e193d74c-a488-4a7b-83d2-866bb9edc44f" (UID: "e193d74c-a488-4a7b-83d2-866bb9edc44f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:45:57.750710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.750523    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:57.750710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.750542    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:57.750710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.750557    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:57.750710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.750571    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:57.750710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.750585    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e193d74c-a488-4a7b-83d2-866bb9edc44f-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:57.751522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.751485    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-podinfo" pod "e193d74c-a488-4a7b-83d2-866bb9edc44f" (UID: "e193d74c-a488-4a7b-83d2-866bb9edc44f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:45:57.751683 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.751666    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-token" pod "e193d74c-a488-4a7b-83d2-866bb9edc44f" (UID: "e193d74c-a488-4a7b-83d2-866bb9edc44f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:57.752425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.752390    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-kube-api-access-7zxbl" pod "e193d74c-a488-4a7b-83d2-866bb9edc44f" (UID: "e193d74c-a488-4a7b-83d2-866bb9edc44f"). InnerVolumeSpecName "kube-api-access-7zxbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:45:57.753208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.753182    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-envoy" pod "e193d74c-a488-4a7b-83d2-866bb9edc44f" (UID: "e193d74c-a488-4a7b-83d2-866bb9edc44f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:45:57.852799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.852766    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:57.852799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.852795    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:57.852799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.852804    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:57.853013 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:57.852813    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zxbl\" (UniqueName: \"kubernetes.io/projected/e193d74c-a488-4a7b-83d2-866bb9edc44f-kube-api-access-7zxbl\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:45:58.341748 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:58.341707    5663 generic.go:356] "Generic (PLEG): container finished" podID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerID="1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46" exitCode=0
Oct 31 12:45:58.341940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:58.341757    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"
Oct 31 12:45:58.341940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:58.341784    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" event={"ID":"e193d74c-a488-4a7b-83d2-866bb9edc44f","Type":"ContainerDied","Data":"1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46"}
Oct 31 12:45:58.341940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:58.341873    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj" event={"ID":"e193d74c-a488-4a7b-83d2-866bb9edc44f","Type":"ContainerDied","Data":"4ac11618288cd9a850955301143a60ecb8d35e1451422e3f6cedbdfda37b0da6"}
Oct 31 12:45:58.341940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:58.341890    5663 scope.go:122] "RemoveContainer" containerID="1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46"
Oct 31 12:45:58.353884 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:58.353855    5663 scope.go:122] "RemoveContainer" containerID="1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46"
Oct 31 12:45:58.354223 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:45:58.354198    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46\": container with ID starting with 1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46 not found: ID does not exist" containerID="1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46"
Oct 31 12:45:58.354308 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:58.354231    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46"} err="failed to get container status \"1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46\": rpc error: code = NotFound desc = could not find container \"1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46\": container with ID starting with 1414bcb444f252b6baa977e1ccb2fcd09464f03319d3d47ebe090c8e44625a46 not found: ID does not exist"
Oct 31 12:45:58.371037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:58.371005    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"]
Oct 31 12:45:58.376239 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:58.376209    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ed25a690-openshift-default-d54cbd5df-vp8xj"]
Oct 31 12:45:59.440743 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:45:59.440709    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" path="/var/lib/kubelet/pods/e193d74c-a488-4a7b-83d2-866bb9edc44f/volumes"
Oct 31 12:46:02.405366 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:02.405256    5663 generic.go:356] "Generic (PLEG): container finished" podID="db35ea34-5527-4982-8a90-3bf05ef57357" containerID="42ae02304d162dfbe913c6a283f7d5c9db63673c03682651c5d70d5b6f5bd2b0" exitCode=0
Oct 31 12:46:02.405366 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:02.405327    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" event={"ID":"db35ea34-5527-4982-8a90-3bf05ef57357","Type":"ContainerDied","Data":"42ae02304d162dfbe913c6a283f7d5c9db63673c03682651c5d70d5b6f5bd2b0"}
Oct 31 12:46:02.405893 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:02.405879    5663 scope.go:122] "RemoveContainer" containerID="42ae02304d162dfbe913c6a283f7d5c9db63673c03682651c5d70d5b6f5bd2b0"
Oct 31 12:46:02.408090 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:02.408047    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:02.408171 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:02.408150    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:02.408240 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:02.408199    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:03.413883 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:03.413848    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" event={"ID":"db35ea34-5527-4982-8a90-3bf05ef57357","Type":"ContainerStarted","Data":"aaec2350b02466a7592b86aefe54a97ea0f2c3323f31ebc34c260fefd92e12ba"}
Oct 31 12:46:04.329809 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:04.329769    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:46:04.385378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:04.381342    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 12:46:04.385378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:04.381435    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 12:46:04.714647 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:04.714552    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:04.734493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:04.734449    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:04.735067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:04.735033    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:04.791518 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:04.791486    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:46:04.842805 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:04.841817    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:46:04.880497 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:04.880453    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:04.908504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:04.908456    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:05.275378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.268455    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 12:46:05.275378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.268517    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 12:46:05.587620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.587468    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"]
Oct 31 12:46:05.588239 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588221    5663 cpu_manager.go:414] "containerMap: removing container" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerName="istio-proxy"
Oct 31 12:46:05.588239 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588238    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerName="istio-proxy"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588251    5663 cpu_manager.go:414] "containerMap: removing container" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588256    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588265    5663 cpu_manager.go:414] "containerMap: removing container" podUID="984090e7-bd56-48d7-933a-a610666c747e" containerName="payload-processing"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588270    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984090e7-bd56-48d7-933a-a610666c747e" containerName="payload-processing"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588283    5663 cpu_manager.go:414] "containerMap: removing container" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588288    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588299    5663 cpu_manager.go:414] "containerMap: removing container" podUID="961f2b49-e1fc-4d28-93f5-880f791be763" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588304    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961f2b49-e1fc-4d28-93f5-880f791be763" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588312    5663 cpu_manager.go:414] "containerMap: removing container" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588316    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588323    5663 cpu_manager.go:414] "containerMap: removing container" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588328    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588334    5663 cpu_manager.go:414] "containerMap: removing container" podUID="961f2b49-e1fc-4d28-93f5-880f791be763" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588338    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961f2b49-e1fc-4d28-93f5-880f791be763" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588349    5663 cpu_manager.go:414] "containerMap: removing container" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588368    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6d164c-805d-4977-b40a-20cc011edfb3" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588384    5663 cpu_manager.go:414] "containerMap: removing container" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588392    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588410    5663 cpu_manager.go:414] "containerMap: removing container" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588416    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588431    5663 cpu_manager.go:414] "containerMap: removing container" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216" containerName="cleanup"
Oct 31 12:46:05.588427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588435    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216" containerName="cleanup"
Oct 31 12:46:05.589535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588559    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="e193d74c-a488-4a7b-83d2-866bb9edc44f" containerName="istio-proxy"
Oct 31 12:46:05.589535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588567    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" containerName="cleanup"
Oct 31 12:46:05.589535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588574    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="984090e7-bd56-48d7-933a-a610666c747e" containerName="payload-processing"
Oct 31 12:46:05.589535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588584    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e" containerName="cleanup"
Oct 31 12:46:05.589535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588591    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="961f2b49-e1fc-4d28-93f5-880f791be763" containerName="cleanup"
Oct 31 12:46:05.589535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588596    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" containerName="cleanup"
Oct 31 12:46:05.589535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588600    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e" containerName="cleanup"
Oct 31 12:46:05.589535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588605    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="961f2b49-e1fc-4d28-93f5-880f791be763" containerName="cleanup"
Oct 31 12:46:05.589535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588610    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98924ce-f10b-45fd-8a83-9f8ebfa424f8" containerName="cleanup"
Oct 31 12:46:05.589535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588616    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216" containerName="cleanup"
Oct 31 12:46:05.589535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.588622    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216" containerName="cleanup"
Oct 31 12:46:05.594037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.591604    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"
Oct 31 12:46:05.594037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.592867    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"]
Oct 31 12:46:05.598058 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.598027    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"]
Oct 31 12:46:05.598929 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.598904    5663 cpu_manager.go:414] "containerMap: removing container" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216" containerName="cleanup"
Oct 31 12:46:05.598929 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.598930    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216" containerName="cleanup"
Oct 31 12:46:05.599080 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.598991    5663 cpu_manager.go:414] "containerMap: removing container" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e" containerName="cleanup"
Oct 31 12:46:05.599080 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.598999    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e" containerName="cleanup"
Oct 31 12:46:05.599080 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.599010    5663 cpu_manager.go:414] "containerMap: removing container" podUID="961f2b49-e1fc-4d28-93f5-880f791be763" containerName="cleanup"
Oct 31 12:46:05.599080 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.599018    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961f2b49-e1fc-4d28-93f5-880f791be763" containerName="cleanup"
Oct 31 12:46:05.599287 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.599198    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f23220-3a1e-457b-9f44-8e64fc4e131e" containerName="cleanup"
Oct 31 12:46:05.599287 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.599215    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="961f2b49-e1fc-4d28-93f5-880f791be763" containerName="cleanup"
Oct 31 12:46:05.599287 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.599233    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="26790f61-8a65-4d3a-8927-c6bb7406b216" containerName="cleanup"
Oct 31 12:46:05.602125 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.602099    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f22e7f\""
Oct 31 12:46:05.603039 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.603018    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f22e7f-dockercfg-b9rjw\""
Oct 31 12:46:05.604494 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.604474    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"
Oct 31 12:46:05.614543 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.614522    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"]
Oct 31 12:46:05.687377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.684090    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k677j\" (UniqueName: \"kubernetes.io/projected/5065822d-b71a-4812-850e-9757c64c4b1a-kube-api-access-k677j\") pod \"payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt\" (UID: \"5065822d-b71a-4812-850e-9757c64c4b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"
Oct 31 12:46:05.687696 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.687631    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5065822d-b71a-4812-850e-9757c64c4b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt\" (UID: \"5065822d-b71a-4812-850e-9757c64c4b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"
Oct 31 12:46:05.687906 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.687893    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxjkl\" (UniqueName: \"kubernetes.io/projected/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-kube-api-access-sxjkl\") pod \"payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr\" (UID: \"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef\") " pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"
Oct 31 12:46:05.688048 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.688039    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr\" (UID: \"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef\") " pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"
Oct 31 12:46:05.789427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.789385    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr\" (UID: \"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef\") " pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"
Oct 31 12:46:05.789859 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.789511    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k677j\" (UniqueName: \"kubernetes.io/projected/5065822d-b71a-4812-850e-9757c64c4b1a-kube-api-access-k677j\") pod \"payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt\" (UID: \"5065822d-b71a-4812-850e-9757c64c4b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"
Oct 31 12:46:05.789859 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.789602    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5065822d-b71a-4812-850e-9757c64c4b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt\" (UID: \"5065822d-b71a-4812-850e-9757c64c4b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"
Oct 31 12:46:05.789859 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.789760    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxjkl\" (UniqueName: \"kubernetes.io/projected/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-kube-api-access-sxjkl\") pod \"payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr\" (UID: \"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef\") " pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"
Oct 31 12:46:05.790911 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.790881    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr\" (UID: \"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef\") " pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"
Oct 31 12:46:05.791192 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.791174    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5065822d-b71a-4812-850e-9757c64c4b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt\" (UID: \"5065822d-b71a-4812-850e-9757c64c4b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"
Oct 31 12:46:05.801912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.801886    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxjkl\" (UniqueName: \"kubernetes.io/projected/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-kube-api-access-sxjkl\") pod \"payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr\" (UID: \"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef\") " pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"
Oct 31 12:46:05.802015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.801887    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k677j\" (UniqueName: \"kubernetes.io/projected/5065822d-b71a-4812-850e-9757c64c4b1a-kube-api-access-k677j\") pod \"payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt\" (UID: \"5065822d-b71a-4812-850e-9757c64c4b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"
Oct 31 12:46:05.917429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.912680    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"
Oct 31 12:46:05.942436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:05.925706    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"
Oct 31 12:46:06.269425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:06.267794    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 12:46:06.269425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:06.267858    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 12:46:06.530517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:06.530484    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"]
Oct 31 12:46:06.532224 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:46:06.532199    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8a6f8d_042d_4b72_8bf8_0a8feac996ef.slice/crio-2456edd99c9a95c7519e556cdeb14a0048a854a97b2b2b958497ae6d74443004 WatchSource:0}: Error finding container 2456edd99c9a95c7519e556cdeb14a0048a854a97b2b2b958497ae6d74443004: Status 404 returned error can't find the container with id 2456edd99c9a95c7519e556cdeb14a0048a854a97b2b2b958497ae6d74443004
Oct 31 12:46:06.591942 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:06.591912    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"]
Oct 31 12:46:06.602080 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:46:06.602054    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5065822d_b71a_4812_850e_9757c64c4b1a.slice/crio-76a9c4d17d18145804f3784448c1c96ad7df00dfeaf339e4ef920479668ba02b WatchSource:0}: Error finding container 76a9c4d17d18145804f3784448c1c96ad7df00dfeaf339e4ef920479668ba02b: Status 404 returned error can't find the container with id 76a9c4d17d18145804f3784448c1c96ad7df00dfeaf339e4ef920479668ba02b
Oct 31 12:46:07.277437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:07.263788    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 12:46:07.277437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:07.263850    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 12:46:07.631069 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:07.631031    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt" event={"ID":"5065822d-b71a-4812-850e-9757c64c4b1a","Type":"ContainerStarted","Data":"b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f"}
Oct 31 12:46:07.631422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:07.631323    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt" event={"ID":"5065822d-b71a-4812-850e-9757c64c4b1a","Type":"ContainerStarted","Data":"76a9c4d17d18145804f3784448c1c96ad7df00dfeaf339e4ef920479668ba02b"}
Oct 31 12:46:07.632659 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:07.632638    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"
Oct 31 12:46:07.635606 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:07.635583    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr" event={"ID":"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef","Type":"ContainerStarted","Data":"df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075"}
Oct 31 12:46:07.635787 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:07.635767    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr" event={"ID":"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef","Type":"ContainerStarted","Data":"2456edd99c9a95c7519e556cdeb14a0048a854a97b2b2b958497ae6d74443004"}
Oct 31 12:46:07.636515 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:07.636497    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"
Oct 31 12:46:07.718415 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:07.715704    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt" podStartSLOduration=2.715657159 podStartE2EDuration="2.715657159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:07.65760868 +0000 UTC m=+1092.398034900" watchObservedRunningTime="2026-10-31 12:46:07.715657159 +0000 UTC m=+1092.456083400"
Oct 31 12:46:07.720986 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:07.720675    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr" podStartSLOduration=2.720655469 podStartE2EDuration="2.720655469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:07.701684476 +0000 UTC m=+1092.442110693" watchObservedRunningTime="2026-10-31 12:46:07.720655469 +0000 UTC m=+1092.461081686"
Oct 31 12:46:08.264874 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:08.264833    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 12:46:08.265161 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:08.265131    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 12:46:09.277095 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:09.277055    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 12:46:09.277785 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:09.277748    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 12:46:10.324972 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:10.324942    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:46:10.326586 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:10.326560    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:46:14.733645 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.733599    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"]
Oct 31 12:46:14.739967 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.739938    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:14.750100 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.750074    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f22e7f-serving-cert\""
Oct 31 12:46:14.753934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.753905    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"]
Oct 31 12:46:14.836198 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.836160    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-maas-api-tls\") pod \"maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:14.836198 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.836200    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06231b8b-0712-4804-a2c2-aa0c3f8bee42-openshift-service-ca\") pod \"maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:14.836480 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.836457    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-metrics-certs\") pod \"maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:14.836669 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.836646    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkctn\" (UniqueName: \"kubernetes.io/projected/06231b8b-0712-4804-a2c2-aa0c3f8bee42-kube-api-access-mkctn\") pod \"maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:14.940960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.940918    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-metrics-certs\") pod \"maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:14.941974 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.941943    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkctn\" (UniqueName: \"kubernetes.io/projected/06231b8b-0712-4804-a2c2-aa0c3f8bee42-kube-api-access-mkctn\") pod \"maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:14.950379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.942179    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-maas-api-tls\") pod \"maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:14.950379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.945432    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06231b8b-0712-4804-a2c2-aa0c3f8bee42-openshift-service-ca\") pod \"maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:14.950379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.946274    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06231b8b-0712-4804-a2c2-aa0c3f8bee42-openshift-service-ca\") pod \"maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:14.973451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.973378    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-metrics-certs\") pod \"maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:14.974329 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.974301    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkctn\" (UniqueName: \"kubernetes.io/projected/06231b8b-0712-4804-a2c2-aa0c3f8bee42-kube-api-access-mkctn\") pod \"maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:14.980443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:14.979850    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-maas-api-tls\") pod \"maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:15.064658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:15.064629    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:15.488929 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:15.488845    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"]
Oct 31 12:46:15.796475 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:15.791796    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q" event={"ID":"06231b8b-0712-4804-a2c2-aa0c3f8bee42","Type":"ContainerStarted","Data":"ba5053d94f6c178e50130aa46cb18797bfa081fd8ced3fe9225134e404919c2c"}
Oct 31 12:46:16.809592 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:16.809551    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q" event={"ID":"06231b8b-0712-4804-a2c2-aa0c3f8bee42","Type":"ContainerStarted","Data":"483823db1f8a79e265f518f0866f57475b33e1173cd01667f3b4ba89fb9a2b19"}
Oct 31 12:46:16.810041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:16.809623    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:16.831067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:16.831014    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q" podStartSLOduration=2.392102534 podStartE2EDuration="2.830999438s" totalImagesPullingTime="438.896904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:16.829267909 +0000 UTC m=+1101.569694127" watchObservedRunningTime="2026-10-31 12:46:16.830999438 +0000 UTC m=+1101.571425698"
Oct 31 12:46:19.655435 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:19.655402    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"
Oct 31 12:46:19.655862 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:19.655497    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"
Oct 31 12:46:22.477261 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:46:22.477225    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:46:22.961383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:22.954633    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:46:31.584025 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:31.583974    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:31.667555 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:31.601505    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:31.670467 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:31.669320    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w" podUID="ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:31.715467 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:31.715422    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:31.750518 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:31.750459    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:46:31.751673 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:31.751641    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:46:31.878891 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:31.878589    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:46:45.947867 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:45.947827    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"]
Oct 31 12:46:45.957704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:45.957667    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:45.963505 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:45.963478    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5c5815-openshift-default-dockercfg-zktkj\""
Oct 31 12:46:45.965374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:45.965334    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"]
Oct 31 12:46:46.121381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.109156    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-podinfo\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.121381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.109214    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-token\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.121381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.109266    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-envoy\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.121381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.109307    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-credential-socket\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.133999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.109344    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istiod-ca-cert\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.133999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.132830    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-data\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.133999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.132991    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-socket\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.133999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.133249    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-certs\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.133999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.133340    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqdj\" (UniqueName: \"kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-kube-api-access-nrqdj\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.235647 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.235495    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqdj\" (UniqueName: \"kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-kube-api-access-nrqdj\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.236023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.235992    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-podinfo\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.236180 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.236160    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-token\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.236331 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.236314    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-envoy\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.236500 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.236482    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-credential-socket\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.236623 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.236608    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istiod-ca-cert\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.236765 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.236751    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-data\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.236952 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.236934    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-socket\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.237254 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.237234    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-certs\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.237893 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.237872    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-certs\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.244379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.238976    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-credential-socket\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.256269 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.244800    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-data\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.256529 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.245421    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istiod-ca-cert\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.256663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.246114    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-socket\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.265844 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.265749    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-envoy\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.274618 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.267790    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-podinfo\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.277166 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.275542    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqdj\" (UniqueName: \"kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-kube-api-access-nrqdj\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.310648 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.310557    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-token\") pod \"e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") " pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:46.592255 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:46.588160    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:47.016631 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.016542    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"]
Oct 31 12:46:47.023479 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:46:47.022769    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e24dd92_1f9a_4bd5_9451_9bc8d05064d0.slice/crio-a33ec74c7d0f72d57d21bb034f60409cc70fe42f52e49b90e708e942fae2ea17 WatchSource:0}: Error finding container a33ec74c7d0f72d57d21bb034f60409cc70fe42f52e49b90e708e942fae2ea17: Status 404 returned error can't find the container with id a33ec74c7d0f72d57d21bb034f60409cc70fe42f52e49b90e708e942fae2ea17
Oct 31 12:46:47.026674 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.026640    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:47.026961 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.026937    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:47.027087 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.027070    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:46:47.199652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.199612    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" event={"ID":"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0","Type":"ContainerStarted","Data":"118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d"}
Oct 31 12:46:47.199871 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.199679    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" event={"ID":"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0","Type":"ContainerStarted","Data":"a33ec74c7d0f72d57d21bb034f60409cc70fe42f52e49b90e708e942fae2ea17"}
Oct 31 12:46:47.224262 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.224212    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" podStartSLOduration=2.22419608 podStartE2EDuration="2.22419608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:47.22150619 +0000 UTC m=+1131.961932422" watchObservedRunningTime="2026-10-31 12:46:47.22419608 +0000 UTC m=+1131.964622297"
Oct 31 12:46:47.592995 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.592957    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:47.656374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.653804    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"]
Oct 31 12:46:47.660033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.660003    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:47.666650 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.665233    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bb3cf5-serving-cert\""
Oct 31 12:46:47.693434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.693284    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"]
Oct 31 12:46:47.721409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.721324    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-maas-api-tls\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:47.721613 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.721594    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-metrics-certs\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:47.721726 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.721709    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0a7567-dc4e-495b-a718-1bc2ca817635-openshift-service-ca\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:47.721848 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.721833    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7676g\" (UniqueName: \"kubernetes.io/projected/6f0a7567-dc4e-495b-a718-1bc2ca817635-kube-api-access-7676g\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:47.826386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.823445    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-metrics-certs\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:47.826386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.823579    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0a7567-dc4e-495b-a718-1bc2ca817635-openshift-service-ca\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:47.826386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.823716    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7676g\" (UniqueName: \"kubernetes.io/projected/6f0a7567-dc4e-495b-a718-1bc2ca817635-kube-api-access-7676g\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:47.826386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.823810    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-maas-api-tls\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:47.826386 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:46:47.824139    5663 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-serving-cert: secret "maas-api-e2e-api-b-bb3cf5-serving-cert" not found
Oct 31 12:46:47.826386 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:46:47.824217    5663 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-maas-api-tls podName:6f0a7567-dc4e-495b-a718-1bc2ca817635 nodeName:}" failed. No retries permitted until 2026-10-31 12:46:48.324197578 +0000 UTC m=+1133.064623772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-maas-api-tls") pod "maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9" (UID: "6f0a7567-dc4e-495b-a718-1bc2ca817635") : secret "maas-api-e2e-api-b-bb3cf5-serving-cert" not found
Oct 31 12:46:47.836377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.827532    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0a7567-dc4e-495b-a718-1bc2ca817635-openshift-service-ca\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:47.856427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.855499    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-metrics-certs\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:47.895270 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:47.895227    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7676g\" (UniqueName: \"kubernetes.io/projected/6f0a7567-dc4e-495b-a718-1bc2ca817635-kube-api-access-7676g\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:48.357993 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:48.357953    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-maas-api-tls\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:48.394397 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:48.390963    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-maas-api-tls\") pod \"maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:48.590725 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:48.590601    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:48.600291 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:48.600253    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:48.908170 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:48.908073    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"]
Oct 31 12:46:49.279880 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:49.279793    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9" event={"ID":"6f0a7567-dc4e-495b-a718-1bc2ca817635","Type":"ContainerStarted","Data":"0ee5236f4f298b09a2d986ae706b8c6e6515f4286a2be17cb10d899d2bc26650"}
Oct 31 12:46:49.280121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:49.280093    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:49.281110 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:49.281090    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:46:50.289605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:50.289564    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9" event={"ID":"6f0a7567-dc4e-495b-a718-1bc2ca817635","Type":"ContainerStarted","Data":"39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182"}
Oct 31 12:46:50.290010 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:50.289766    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:50.314348 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:50.314297    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9" podStartSLOduration=2.846888848 podStartE2EDuration="3.314280605s" totalImagesPullingTime="467.391757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:50.309317317 +0000 UTC m=+1135.049743532" watchObservedRunningTime="2026-10-31 12:46:50.314280605 +0000 UTC m=+1135.054706821"
Oct 31 12:46:50.957870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:50.957835    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"]
Oct 31 12:46:50.962629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:50.962612    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"
Oct 31 12:46:50.968754 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:50.968731    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5c5815\""
Oct 31 12:46:50.970180 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:50.970159    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5c5815-dockercfg-ht77f\""
Oct 31 12:46:50.975163 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:50.975142    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"]
Oct 31 12:46:50.993466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:50.993436    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dwfm\" (UniqueName: \"kubernetes.io/projected/119e2bed-7181-48bc-a8f2-8be78a229cee-kube-api-access-7dwfm\") pod \"payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd\" (UID: \"119e2bed-7181-48bc-a8f2-8be78a229cee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"
Oct 31 12:46:50.993636 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:50.993483    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd\" (UID: \"119e2bed-7181-48bc-a8f2-8be78a229cee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"
Oct 31 12:46:51.146379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:51.114141    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwfm\" (UniqueName: \"kubernetes.io/projected/119e2bed-7181-48bc-a8f2-8be78a229cee-kube-api-access-7dwfm\") pod \"payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd\" (UID: \"119e2bed-7181-48bc-a8f2-8be78a229cee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"
Oct 31 12:46:51.146379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:51.140455    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd\" (UID: \"119e2bed-7181-48bc-a8f2-8be78a229cee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"
Oct 31 12:46:51.146379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:51.141635    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd\" (UID: \"119e2bed-7181-48bc-a8f2-8be78a229cee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"
Oct 31 12:46:51.225436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:51.212602    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dwfm\" (UniqueName: \"kubernetes.io/projected/119e2bed-7181-48bc-a8f2-8be78a229cee-kube-api-access-7dwfm\") pod \"payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd\" (UID: \"119e2bed-7181-48bc-a8f2-8be78a229cee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"
Oct 31 12:46:51.290413 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:51.288709    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"
Oct 31 12:46:51.751411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:51.751380    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"]
Oct 31 12:46:51.751951 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:46:51.751925    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119e2bed_7181_48bc_a8f2_8be78a229cee.slice/crio-20d3ee4031adaa7f9c7d8390c84586ab951ca319bc3cb39335a98abc14e8d727 WatchSource:0}: Error finding container 20d3ee4031adaa7f9c7d8390c84586ab951ca319bc3cb39335a98abc14e8d727: Status 404 returned error can't find the container with id 20d3ee4031adaa7f9c7d8390c84586ab951ca319bc3cb39335a98abc14e8d727
Oct 31 12:46:52.450928 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:52.450823    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd" event={"ID":"119e2bed-7181-48bc-a8f2-8be78a229cee","Type":"ContainerStarted","Data":"aae2c5e84066418a6043a50d9cd7186e27e120ecf7037bb4c00cebcfc9357530"}
Oct 31 12:46:52.450928 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:52.450889    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd" event={"ID":"119e2bed-7181-48bc-a8f2-8be78a229cee","Type":"ContainerStarted","Data":"20d3ee4031adaa7f9c7d8390c84586ab951ca319bc3cb39335a98abc14e8d727"}
Oct 31 12:46:52.451415 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:52.450937    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"
Oct 31 12:46:52.472551 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:52.472501    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd" podStartSLOduration=2.472484115 podStartE2EDuration="2.472484115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:46:52.46839932 +0000 UTC m=+1137.208825534" watchObservedRunningTime="2026-10-31 12:46:52.472484115 +0000 UTC m=+1137.212910348"
Oct 31 12:46:52.478293 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:46:52.478248    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:46:56.461516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:56.461485    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:46:58.674859 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:58.674828    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"]
Oct 31 12:46:58.680691 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:58.680667    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"
Oct 31 12:46:58.700762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:58.699632    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"]
Oct 31 12:46:58.703633 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:58.703608    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-fb2b1d\""
Oct 31 12:46:58.784670 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:58.784621    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk\" (UID: \"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"
Oct 31 12:46:58.785048 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:58.785017    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcbmm\" (UniqueName: \"kubernetes.io/projected/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-kube-api-access-dcbmm\") pod \"payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk\" (UID: \"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"
Oct 31 12:46:58.898376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:58.894956    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk\" (UID: \"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"
Oct 31 12:46:58.898376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:58.895295    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcbmm\" (UniqueName: \"kubernetes.io/projected/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-kube-api-access-dcbmm\") pod \"payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk\" (UID: \"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"
Oct 31 12:46:58.898376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:58.896517    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk\" (UID: \"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"
Oct 31 12:46:58.968429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:58.962214    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcbmm\" (UniqueName: \"kubernetes.io/projected/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-kube-api-access-dcbmm\") pod \"payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk\" (UID: \"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"
Oct 31 12:46:59.005001 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:59.004704    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"
Oct 31 12:46:59.604519 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:59.604481    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"]
Oct 31 12:46:59.604752 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:46:59.604725    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca75a2f1_e93d_47ca_a966_1bd1ad9514a5.slice/crio-3416f04c54ef9a1b97c96e166a74ffd896bb37ace52150bd481f15cdd624af3a WatchSource:0}: Error finding container 3416f04c54ef9a1b97c96e166a74ffd896bb37ace52150bd481f15cdd624af3a: Status 404 returned error can't find the container with id 3416f04c54ef9a1b97c96e166a74ffd896bb37ace52150bd481f15cdd624af3a
Oct 31 12:46:59.699791 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:46:59.699764    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk" event={"ID":"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5","Type":"ContainerStarted","Data":"3416f04c54ef9a1b97c96e166a74ffd896bb37ace52150bd481f15cdd624af3a"}
Oct 31 12:47:00.708451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:00.708412    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk" event={"ID":"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5","Type":"ContainerStarted","Data":"cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b"}
Oct 31 12:47:00.708861 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:00.708603    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"
Oct 31 12:47:00.730000 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:00.729949    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk" podStartSLOduration=2.729931164 podStartE2EDuration="2.729931164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:00.726210031 +0000 UTC m=+1145.466636247" watchObservedRunningTime="2026-10-31 12:47:00.729931164 +0000 UTC m=+1145.470357411"
Oct 31 12:47:03.563913 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:03.563884    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"
Oct 31 12:47:03.803482 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:03.803446    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7029-c7e591"]
Oct 31 12:47:03.808243 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:03.808221    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-c7e591"
Oct 31 12:47:03.812035 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:03.812008    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-klvx2\""
Oct 31 12:47:03.815275 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:03.815223    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7029-c7e591"]
Oct 31 12:47:03.891633 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:03.891598    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6xx2\" (UniqueName: \"kubernetes.io/projected/679a82ec-b01e-4ac2-93a0-13ceaa6a88db-kube-api-access-m6xx2\") pod \"test-curl-7029-c7e591\" (UID: \"679a82ec-b01e-4ac2-93a0-13ceaa6a88db\") " pod="opendatahub/test-curl-7029-c7e591"
Oct 31 12:47:04.001400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:04.000781    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6xx2\" (UniqueName: \"kubernetes.io/projected/679a82ec-b01e-4ac2-93a0-13ceaa6a88db-kube-api-access-m6xx2\") pod \"test-curl-7029-c7e591\" (UID: \"679a82ec-b01e-4ac2-93a0-13ceaa6a88db\") " pod="opendatahub/test-curl-7029-c7e591"
Oct 31 12:47:04.029022 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:04.028986    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6xx2\" (UniqueName: \"kubernetes.io/projected/679a82ec-b01e-4ac2-93a0-13ceaa6a88db-kube-api-access-m6xx2\") pod \"test-curl-7029-c7e591\" (UID: \"679a82ec-b01e-4ac2-93a0-13ceaa6a88db\") " pod="opendatahub/test-curl-7029-c7e591"
Oct 31 12:47:04.146433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:04.137218    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-c7e591"
Oct 31 12:47:04.784415 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:04.784163    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7029-c7e591"]
Oct 31 12:47:04.785998 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:47:04.785810    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod679a82ec_b01e_4ac2_93a0_13ceaa6a88db.slice/crio-0601f41f65610ca7f84cab86ea08ebee078d5e2b9221e8ee5c3f85636b7c2e52 WatchSource:0}: Error finding container 0601f41f65610ca7f84cab86ea08ebee078d5e2b9221e8ee5c3f85636b7c2e52: Status 404 returned error can't find the container with id 0601f41f65610ca7f84cab86ea08ebee078d5e2b9221e8ee5c3f85636b7c2e52
Oct 31 12:47:05.764290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:05.764259    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7029-c7e591" event={"ID":"679a82ec-b01e-4ac2-93a0-13ceaa6a88db","Type":"ContainerStarted","Data":"0145fda551610b4e4fae88ac34ab52770f06c96fe5146c405277969e4a53196d"}
Oct 31 12:47:05.764491 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:05.764301    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7029-c7e591" event={"ID":"679a82ec-b01e-4ac2-93a0-13ceaa6a88db","Type":"ContainerStarted","Data":"0601f41f65610ca7f84cab86ea08ebee078d5e2b9221e8ee5c3f85636b7c2e52"}
Oct 31 12:47:05.783318 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:05.783263    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7029-c7e591" podStartSLOduration=2.783250013 podStartE2EDuration="2.783250013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:05.779208893 +0000 UTC m=+1150.519635109" watchObservedRunningTime="2026-10-31 12:47:05.783250013 +0000 UTC m=+1150.523676229"
Oct 31 12:47:06.933559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:06.933477    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7029-c7e591"]
Oct 31 12:47:06.976300 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:06.976256    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7029-c7e591"]
Oct 31 12:47:07.780095 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:07.780053    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7029-c7e591" podUID="679a82ec-b01e-4ac2-93a0-13ceaa6a88db" containerName="test-curl-7029-c7e591" containerID="cri-o://0145fda551610b4e4fae88ac34ab52770f06c96fe5146c405277969e4a53196d" gracePeriod=2
Oct 31 12:47:09.997458 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:09.997432    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-c7e591"
Oct 31 12:47:10.049655 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.049564    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679a82ec-b01e-4ac2-93a0-13ceaa6a88db-kube-api-access-m6xx2\" (UniqueName: \"kubernetes.io/projected/679a82ec-b01e-4ac2-93a0-13ceaa6a88db-kube-api-access-m6xx2\") pod \"679a82ec-b01e-4ac2-93a0-13ceaa6a88db\" (UID: \"679a82ec-b01e-4ac2-93a0-13ceaa6a88db\") "
Oct 31 12:47:10.051676 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.051653    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679a82ec-b01e-4ac2-93a0-13ceaa6a88db-kube-api-access-m6xx2" pod "679a82ec-b01e-4ac2-93a0-13ceaa6a88db" (UID: "679a82ec-b01e-4ac2-93a0-13ceaa6a88db"). InnerVolumeSpecName "kube-api-access-m6xx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:10.154383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.154249    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6xx2\" (UniqueName: \"kubernetes.io/projected/679a82ec-b01e-4ac2-93a0-13ceaa6a88db-kube-api-access-m6xx2\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:10.668217 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.668184    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"]
Oct 31 12:47:10.668863 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.668849    5663 cpu_manager.go:414] "containerMap: removing container" podUID="679a82ec-b01e-4ac2-93a0-13ceaa6a88db" containerName="test-curl-7029-c7e591"
Oct 31 12:47:10.668916 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.668864    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679a82ec-b01e-4ac2-93a0-13ceaa6a88db" containerName="test-curl-7029-c7e591"
Oct 31 12:47:10.669014 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.669005    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="679a82ec-b01e-4ac2-93a0-13ceaa6a88db" containerName="test-curl-7029-c7e591"
Oct 31 12:47:10.673041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.673015    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:10.691399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.689889    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"]
Oct 31 12:47:10.734517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.734478    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-fb2b1d-serving-cert\""
Oct 31 12:47:10.765156 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.765122    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6lh\" (UniqueName: \"kubernetes.io/projected/fc521967-8ba8-4074-bf02-9e7ebf169601-kube-api-access-8c6lh\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:10.765313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.765258    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:10.765415 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.765323    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:10.765552 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.765539    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc521967-8ba8-4074-bf02-9e7ebf169601-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:10.806494 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.806458    5663 generic.go:356] "Generic (PLEG): container finished" podID="679a82ec-b01e-4ac2-93a0-13ceaa6a88db" containerID="0145fda551610b4e4fae88ac34ab52770f06c96fe5146c405277969e4a53196d" exitCode=137
Oct 31 12:47:10.806678 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.806511    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-c7e591"
Oct 31 12:47:10.806678 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.806559    5663 scope.go:122] "RemoveContainer" containerID="0145fda551610b4e4fae88ac34ab52770f06c96fe5146c405277969e4a53196d"
Oct 31 12:47:10.817611 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.817591    5663 scope.go:122] "RemoveContainer" containerID="0145fda551610b4e4fae88ac34ab52770f06c96fe5146c405277969e4a53196d"
Oct 31 12:47:10.817910 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:47:10.817893    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0145fda551610b4e4fae88ac34ab52770f06c96fe5146c405277969e4a53196d\": container with ID starting with 0145fda551610b4e4fae88ac34ab52770f06c96fe5146c405277969e4a53196d not found: ID does not exist" containerID="0145fda551610b4e4fae88ac34ab52770f06c96fe5146c405277969e4a53196d"
Oct 31 12:47:10.817954 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.817919    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0145fda551610b4e4fae88ac34ab52770f06c96fe5146c405277969e4a53196d"} err="failed to get container status \"0145fda551610b4e4fae88ac34ab52770f06c96fe5146c405277969e4a53196d\": rpc error: code = NotFound desc = could not find container \"0145fda551610b4e4fae88ac34ab52770f06c96fe5146c405277969e4a53196d\": container with ID starting with 0145fda551610b4e4fae88ac34ab52770f06c96fe5146c405277969e4a53196d not found: ID does not exist"
Oct 31 12:47:10.818960 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:47:10.818939    5663 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7029-c7e591\" is forbidden: User \"system:node:ip-10-0-140-137.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-137.us-east-2.compute.internal' and this object" podUID="679a82ec-b01e-4ac2-93a0-13ceaa6a88db" pod="opendatahub/test-curl-7029-c7e591"
Oct 31 12:47:10.866816 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.866790    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc521967-8ba8-4074-bf02-9e7ebf169601-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:10.866948 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.866895    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6lh\" (UniqueName: \"kubernetes.io/projected/fc521967-8ba8-4074-bf02-9e7ebf169601-kube-api-access-8c6lh\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:10.867007 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.866981    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:10.867087 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.867031    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:10.867147 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:47:10.867116    5663 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-serving-cert: secret "maas-api-e2e-shared-a-w2-fb2b1d-serving-cert" not found
Oct 31 12:47:10.867208 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:47:10.867192    5663 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-maas-api-tls podName:fc521967-8ba8-4074-bf02-9e7ebf169601 nodeName:}" failed. No retries permitted until 2026-10-31 12:47:11.367169801 +0000 UTC m=+1156.107596012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-maas-api-tls") pod "maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct" (UID: "fc521967-8ba8-4074-bf02-9e7ebf169601") : secret "maas-api-e2e-shared-a-w2-fb2b1d-serving-cert" not found
Oct 31 12:47:10.867538 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.867518    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc521967-8ba8-4074-bf02-9e7ebf169601-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:10.869477 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.869460    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:10.879332 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:10.879300    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6lh\" (UniqueName: \"kubernetes.io/projected/fc521967-8ba8-4074-bf02-9e7ebf169601-kube-api-access-8c6lh\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:11.389378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:11.385788    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:11.406404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:11.390397    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:11.511931 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:11.511891    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679a82ec-b01e-4ac2-93a0-13ceaa6a88db" path="/var/lib/kubelet/pods/679a82ec-b01e-4ac2-93a0-13ceaa6a88db/volumes"
Oct 31 12:47:11.601377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:11.595691    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:11.739378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:11.729343    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"
Oct 31 12:47:12.044449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:12.044422    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"]
Oct 31 12:47:12.045531 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:47:12.045506    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc521967_8ba8_4074_bf02_9e7ebf169601.slice/crio-7cf04b02a261779a6368d816e50864b60e313145f8c08a38d526ebaef4526f27 WatchSource:0}: Error finding container 7cf04b02a261779a6368d816e50864b60e313145f8c08a38d526ebaef4526f27: Status 404 returned error can't find the container with id 7cf04b02a261779a6368d816e50864b60e313145f8c08a38d526ebaef4526f27
Oct 31 12:47:12.835945 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:12.835905    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct" event={"ID":"fc521967-8ba8-4074-bf02-9e7ebf169601","Type":"ContainerStarted","Data":"88c227e38b50c709c24ad55613bef3d2b64ed7c9ad206986ec2a594cbc93640f"}
Oct 31 12:47:12.836396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:12.835972    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct" event={"ID":"fc521967-8ba8-4074-bf02-9e7ebf169601","Type":"ContainerStarted","Data":"7cf04b02a261779a6368d816e50864b60e313145f8c08a38d526ebaef4526f27"}
Oct 31 12:47:12.836396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:12.836123    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:12.858547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:12.858494    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct" podStartSLOduration=2.424017638 podStartE2EDuration="2.858479783s" totalImagesPullingTime="434.462145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:12.856714448 +0000 UTC m=+1157.597140665" watchObservedRunningTime="2026-10-31 12:47:12.858479783 +0000 UTC m=+1157.598905999"
Oct 31 12:47:17.666567 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:17.666517    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"]
Oct 31 12:47:17.677648 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:17.677619    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"]
Oct 31 12:47:17.955366 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:17.955264    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"]
Oct 31 12:47:17.976222 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:17.976187    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"]
Oct 31 12:47:18.904430 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:18.898641    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:47:22.508966 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:47:22.508925    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:47:22.667418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:22.667326    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs" podUID="89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1" containerName="payload-processing" containerID="cri-o://c4eae499a7bbbbc0b7c7031790d0ae8511d5027ca69ac989118b000b8c39d53f" gracePeriod=25
Oct 31 12:47:22.678831 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:22.678800    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj" podUID="af38a90f-5827-452c-b245-7e147b5983eb" containerName="payload-pre-processing" containerID="cri-o://ec1e82dc1906eeb6f65112e520bd1cb60952c9f883ce448e8068b3d2609faf1b" gracePeriod=25
Oct 31 12:47:22.955893 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:22.955848    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr" podUID="6c8a6f8d-042d-4b72-8bf8-0a8feac996ef" containerName="payload-processing" containerID="cri-o://df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075" gracePeriod=25
Oct 31 12:47:22.977110 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:22.977071    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt" podUID="5065822d-b71a-4812-850e-9757c64c4b1a" containerName="payload-pre-processing" containerID="cri-o://b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f" gracePeriod=25
Oct 31 12:47:23.413907 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.413879    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"
Oct 31 12:47:23.496002 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.495969    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"
Oct 31 12:47:23.531805 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.531715    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-kube-api-access-sxjkl\" (UniqueName: \"kubernetes.io/projected/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-kube-api-access-sxjkl\") pod \"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef\" (UID: \"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef\") "
Oct 31 12:47:23.532498 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.532473    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-plugins-config-volume\") pod \"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef\" (UID: \"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef\") "
Oct 31 12:47:23.532987 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.532960    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-plugins-config-volume" pod "6c8a6f8d-042d-4b72-8bf8-0a8feac996ef" (UID: "6c8a6f8d-042d-4b72-8bf8-0a8feac996ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:47:23.535046 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.535019    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:23.536301 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.536270    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-kube-api-access-sxjkl" pod "6c8a6f8d-042d-4b72-8bf8-0a8feac996ef" (UID: "6c8a6f8d-042d-4b72-8bf8-0a8feac996ef"). InnerVolumeSpecName "kube-api-access-sxjkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:23.636279 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.636247    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5065822d-b71a-4812-850e-9757c64c4b1a-kube-api-access-k677j\" (UniqueName: \"kubernetes.io/projected/5065822d-b71a-4812-850e-9757c64c4b1a-kube-api-access-k677j\") pod \"5065822d-b71a-4812-850e-9757c64c4b1a\" (UID: \"5065822d-b71a-4812-850e-9757c64c4b1a\") "
Oct 31 12:47:23.636521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.636508    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5065822d-b71a-4812-850e-9757c64c4b1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5065822d-b71a-4812-850e-9757c64c4b1a-plugins-config-volume\") pod \"5065822d-b71a-4812-850e-9757c64c4b1a\" (UID: \"5065822d-b71a-4812-850e-9757c64c4b1a\") "
Oct 31 12:47:23.636869 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.636842    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5065822d-b71a-4812-850e-9757c64c4b1a-plugins-config-volume" pod "5065822d-b71a-4812-850e-9757c64c4b1a" (UID: "5065822d-b71a-4812-850e-9757c64c4b1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:47:23.637050 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.637031    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxjkl\" (UniqueName: \"kubernetes.io/projected/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef-kube-api-access-sxjkl\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:23.637127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.637055    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5065822d-b71a-4812-850e-9757c64c4b1a-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:23.638452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.638426    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5065822d-b71a-4812-850e-9757c64c4b1a-kube-api-access-k677j" pod "5065822d-b71a-4812-850e-9757c64c4b1a" (UID: "5065822d-b71a-4812-850e-9757c64c4b1a"). InnerVolumeSpecName "kube-api-access-k677j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:23.738672 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:23.738639    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k677j\" (UniqueName: \"kubernetes.io/projected/5065822d-b71a-4812-850e-9757c64c4b1a-kube-api-access-k677j\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:24.045967 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.045928    5663 generic.go:356] "Generic (PLEG): container finished" podID="5065822d-b71a-4812-850e-9757c64c4b1a" containerID="b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f" exitCode=0
Oct 31 12:47:24.046169 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.045981    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"
Oct 31 12:47:24.046169 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.046012    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt" event={"ID":"5065822d-b71a-4812-850e-9757c64c4b1a","Type":"ContainerDied","Data":"b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f"}
Oct 31 12:47:24.046169 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.046066    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt" event={"ID":"5065822d-b71a-4812-850e-9757c64c4b1a","Type":"ContainerDied","Data":"76a9c4d17d18145804f3784448c1c96ad7df00dfeaf339e4ef920479668ba02b"}
Oct 31 12:47:24.046169 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.046090    5663 scope.go:122] "RemoveContainer" containerID="b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f"
Oct 31 12:47:24.047584 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.047565    5663 generic.go:356] "Generic (PLEG): container finished" podID="6c8a6f8d-042d-4b72-8bf8-0a8feac996ef" containerID="df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075" exitCode=0
Oct 31 12:47:24.047674 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.047596    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr" event={"ID":"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef","Type":"ContainerDied","Data":"df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075"}
Oct 31 12:47:24.047674 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.047646    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"
Oct 31 12:47:24.047674 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.047663    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr" event={"ID":"6c8a6f8d-042d-4b72-8bf8-0a8feac996ef","Type":"ContainerDied","Data":"2456edd99c9a95c7519e556cdeb14a0048a854a97b2b2b958497ae6d74443004"}
Oct 31 12:47:24.057444 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.057416    5663 scope.go:122] "RemoveContainer" containerID="b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f"
Oct 31 12:47:24.057789 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:47:24.057770    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f\": container with ID starting with b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f not found: ID does not exist" containerID="b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f"
Oct 31 12:47:24.057846 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.057797    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f"} err="failed to get container status \"b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f\": rpc error: code = NotFound desc = could not find container \"b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f\": container with ID starting with b73f1a38bf79ff2bd100826ccb6310b4d9be0e045535581e062ec0a5af961f5f not found: ID does not exist"
Oct 31 12:47:24.057846 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.057816    5663 scope.go:122] "RemoveContainer" containerID="df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075"
Oct 31 12:47:24.067996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.067972    5663 scope.go:122] "RemoveContainer" containerID="df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075"
Oct 31 12:47:24.068342 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:47:24.068320    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075\": container with ID starting with df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075 not found: ID does not exist" containerID="df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075"
Oct 31 12:47:24.068454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.068396    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075"} err="failed to get container status \"df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075\": rpc error: code = NotFound desc = could not find container \"df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075\": container with ID starting with df0d41c3552f6598341f1d3b5252c07f353f758e2d26192b826715d7d6987075 not found: ID does not exist"
Oct 31 12:47:24.078489 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.078465    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"]
Oct 31 12:47:24.084125 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.084103    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f22e7f-898dfbf47-7d7lt"]
Oct 31 12:47:24.094773 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.094750    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"]
Oct 31 12:47:24.099977 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:24.099953    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f22e7f-685685b855-xjhxr"]
Oct 31 12:47:25.441282 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:25.441252    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5065822d-b71a-4812-850e-9757c64c4b1a" path="/var/lib/kubelet/pods/5065822d-b71a-4812-850e-9757c64c4b1a/volumes"
Oct 31 12:47:25.441839 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:25.441821    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8a6f8d-042d-4b72-8bf8-0a8feac996ef" path="/var/lib/kubelet/pods/6c8a6f8d-042d-4b72-8bf8-0a8feac996ef/volumes"
Oct 31 12:47:29.705943 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.705904    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"]
Oct 31 12:47:29.707061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.707033    5663 cpu_manager.go:414] "containerMap: removing container" podUID="6c8a6f8d-042d-4b72-8bf8-0a8feac996ef" containerName="payload-processing"
Oct 31 12:47:29.707061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.707061    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8a6f8d-042d-4b72-8bf8-0a8feac996ef" containerName="payload-processing"
Oct 31 12:47:29.707256 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.707134    5663 cpu_manager.go:414] "containerMap: removing container" podUID="5065822d-b71a-4812-850e-9757c64c4b1a" containerName="payload-pre-processing"
Oct 31 12:47:29.707256 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.707143    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5065822d-b71a-4812-850e-9757c64c4b1a" containerName="payload-pre-processing"
Oct 31 12:47:29.707413 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.707286    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8a6f8d-042d-4b72-8bf8-0a8feac996ef" containerName="payload-processing"
Oct 31 12:47:29.707413 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.707308    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="5065822d-b71a-4812-850e-9757c64c4b1a" containerName="payload-pre-processing"
Oct 31 12:47:29.725706 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.723291    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.727557 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.727530    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-015701-openshift-default-dockercfg-prb4x\""
Oct 31 12:47:29.727803 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.727775    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"]
Oct 31 12:47:29.866757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.866709    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e10166-45ce-4baa-97b9-6b1544701e43-istio-podinfo\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.866952 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.866927    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-envoy\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.867013 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.866966    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-certs\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.867013 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.867003    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-data\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.867109 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.867037    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e10166-45ce-4baa-97b9-6b1544701e43-istiod-ca-cert\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.867854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.867272    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-istio-token\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.877420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.871971    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dzjt\" (UniqueName: \"kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-kube-api-access-6dzjt\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.877420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.872316    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-credential-socket\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.877420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.872433    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-socket\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.980426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.979807    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-envoy\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.980426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.979865    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-certs\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.980426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.979902    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-data\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.980426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.979931    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e10166-45ce-4baa-97b9-6b1544701e43-istiod-ca-cert\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.980426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.980066    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-istio-token\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.980426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.980167    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dzjt\" (UniqueName: \"kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-kube-api-access-6dzjt\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.980426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.980246    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-credential-socket\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.980426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.980300    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-socket\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.981007 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.980473    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e10166-45ce-4baa-97b9-6b1544701e43-istio-podinfo\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:29.982449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.982245    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e10166-45ce-4baa-97b9-6b1544701e43-istiod-ca-cert\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:30.000422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.997527    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e10166-45ce-4baa-97b9-6b1544701e43-istio-podinfo\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:30.000422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.998482    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-credential-socket\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:30.000422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.998704    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-socket\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:30.000422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.998812    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-certs\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:30.000422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:29.998971    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-data\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:30.031379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:30.027667    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-envoy\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:30.065374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:30.048907    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-istio-token\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:30.065374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:30.055070    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dzjt\" (UniqueName: \"kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-kube-api-access-6dzjt\") pod \"e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") " pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:30.076378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:30.065688    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:30.896298 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:30.896265    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"]
Oct 31 12:47:30.897672 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:47:30.897643    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e10166_45ce_4baa_97b9_6b1544701e43.slice/crio-7d73e5d0b4e42d76f2164d9ba719c9172037824eca1837eee8f0838e952c61d8 WatchSource:0}: Error finding container 7d73e5d0b4e42d76f2164d9ba719c9172037824eca1837eee8f0838e952c61d8: Status 404 returned error can't find the container with id 7d73e5d0b4e42d76f2164d9ba719c9172037824eca1837eee8f0838e952c61d8
Oct 31 12:47:30.900618 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:30.900575    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:47:30.900744 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:30.900676    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:47:30.900810 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:30.900757    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:47:31.304256 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:31.304156    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj" podUID="af38a90f-5827-452c-b245-7e147b5983eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:47:31.440116 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:31.440072    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" event={"ID":"22e10166-45ce-4baa-97b9-6b1544701e43","Type":"ContainerStarted","Data":"06624e83e2d7d3ec10dda92fc212e3731fe48d9f34fd1cb2b767bf0b148c6150"}
Oct 31 12:47:31.440284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:31.440131    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" event={"ID":"22e10166-45ce-4baa-97b9-6b1544701e43","Type":"ContainerStarted","Data":"7d73e5d0b4e42d76f2164d9ba719c9172037824eca1837eee8f0838e952c61d8"}
Oct 31 12:47:31.469892 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:31.469828    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" podStartSLOduration=2.4698094250000002 podStartE2EDuration="2.469809425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:31.464514082 +0000 UTC m=+1176.204940331" watchObservedRunningTime="2026-10-31 12:47:31.469809425 +0000 UTC m=+1176.210235642"
Oct 31 12:47:32.067065 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.067031    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:32.073533 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.073502    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:32.168447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.168412    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"]
Oct 31 12:47:32.173718 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.173694    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"
Oct 31 12:47:32.179106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.179065    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-787316-dockercfg-s8bkn\""
Oct 31 12:47:32.182649 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.182615    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-787316\""
Oct 31 12:47:32.183792 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.183765    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"]
Oct 31 12:47:32.416471 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.365220    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-787316-65b96d4df-vv7rg\" (UID: \"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9\") " pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"
Oct 31 12:47:32.416941 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.416910    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn9vj\" (UniqueName: \"kubernetes.io/projected/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-kube-api-access-cn9vj\") pod \"payload-processing-e2e-api-b-787316-65b96d4df-vv7rg\" (UID: \"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9\") " pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"
Oct 31 12:47:32.546163 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.546120    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-787316-65b96d4df-vv7rg\" (UID: \"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9\") " pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"
Oct 31 12:47:32.546368 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.546331    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn9vj\" (UniqueName: \"kubernetes.io/projected/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-kube-api-access-cn9vj\") pod \"payload-processing-e2e-api-b-787316-65b96d4df-vv7rg\" (UID: \"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9\") " pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"
Oct 31 12:47:32.603381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.597707    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-787316-65b96d4df-vv7rg\" (UID: \"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9\") " pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"
Oct 31 12:47:32.649388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.641541    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs" podUID="89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:47:32.730439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.707597    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:32.731161 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.730794    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn9vj\" (UniqueName: \"kubernetes.io/projected/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-kube-api-access-cn9vj\") pod \"payload-processing-e2e-api-b-787316-65b96d4df-vv7rg\" (UID: \"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9\") " pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"
Oct 31 12:47:32.772297 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.771863    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:47:32.799317 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:32.797957    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"
Oct 31 12:47:33.547433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:33.547400    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"]
Oct 31 12:47:33.548067 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:47:33.548041    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40fca5d_2ca4_4ea7_9e33_59dfe670cbb9.slice/crio-59aa9b87fbf4f5c47b51bc552eb8027d9954f510362db958f126b73a9b43917f WatchSource:0}: Error finding container 59aa9b87fbf4f5c47b51bc552eb8027d9954f510362db958f126b73a9b43917f: Status 404 returned error can't find the container with id 59aa9b87fbf4f5c47b51bc552eb8027d9954f510362db958f126b73a9b43917f
Oct 31 12:47:33.717210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:33.717167    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg" event={"ID":"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9","Type":"ContainerStarted","Data":"5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d"}
Oct 31 12:47:33.717472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:33.717236    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg" event={"ID":"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9","Type":"ContainerStarted","Data":"59aa9b87fbf4f5c47b51bc552eb8027d9954f510362db958f126b73a9b43917f"}
Oct 31 12:47:33.717472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:33.717378    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"
Oct 31 12:47:33.737797 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:33.737743    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg" podStartSLOduration=1.737724042 podStartE2EDuration="1.737724042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:33.737297178 +0000 UTC m=+1178.477723394" watchObservedRunningTime="2026-10-31 12:47:33.737724042 +0000 UTC m=+1178.478150259"
Oct 31 12:47:35.069123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:35.069089    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"]
Oct 31 12:47:35.074363 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:35.074323    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"
Oct 31 12:47:35.090629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:35.079998    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d4dd68-dockercfg-j6f8t\""
Oct 31 12:47:35.090629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:35.085500    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"]
Oct 31 12:47:35.096506 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:35.096475    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d4dd68\""
Oct 31 12:47:35.127108 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:35.127071    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca29716-d6b0-4d20-8488-556d6018aefe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg\" (UID: \"aca29716-d6b0-4d20-8488-556d6018aefe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"
Oct 31 12:47:35.127296 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:35.127218    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hnm6\" (UniqueName: \"kubernetes.io/projected/aca29716-d6b0-4d20-8488-556d6018aefe-kube-api-access-8hnm6\") pod \"payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg\" (UID: \"aca29716-d6b0-4d20-8488-556d6018aefe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"
Oct 31 12:47:35.233380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:35.232590    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca29716-d6b0-4d20-8488-556d6018aefe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg\" (UID: \"aca29716-d6b0-4d20-8488-556d6018aefe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"
Oct 31 12:47:35.233380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:35.232750    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hnm6\" (UniqueName: \"kubernetes.io/projected/aca29716-d6b0-4d20-8488-556d6018aefe-kube-api-access-8hnm6\") pod \"payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg\" (UID: \"aca29716-d6b0-4d20-8488-556d6018aefe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"
Oct 31 12:47:35.237378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:35.234267    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca29716-d6b0-4d20-8488-556d6018aefe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg\" (UID: \"aca29716-d6b0-4d20-8488-556d6018aefe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"
Oct 31 12:47:35.275377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:35.266729    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hnm6\" (UniqueName: \"kubernetes.io/projected/aca29716-d6b0-4d20-8488-556d6018aefe-kube-api-access-8hnm6\") pod \"payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg\" (UID: \"aca29716-d6b0-4d20-8488-556d6018aefe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"
Oct 31 12:47:35.395372 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:35.393634    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"
Oct 31 12:47:36.025424 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.025388    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"]
Oct 31 12:47:36.026830 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:47:36.026803    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca29716_d6b0_4d20_8488_556d6018aefe.slice/crio-77c7cbca1bfd7560ce61f4c0651a631d792f4a3f8aab89063b7bd47e85fe467a WatchSource:0}: Error finding container 77c7cbca1bfd7560ce61f4c0651a631d792f4a3f8aab89063b7bd47e85fe467a: Status 404 returned error can't find the container with id 77c7cbca1bfd7560ce61f4c0651a631d792f4a3f8aab89063b7bd47e85fe467a
Oct 31 12:47:36.272100 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.272060    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"]
Oct 31 12:47:36.277262 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.277239    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"
Oct 31 12:47:36.284956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.284927    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"]
Oct 31 12:47:36.329079 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.328314    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-015701-dockercfg-hz68c\""
Oct 31 12:47:36.381808 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.379629    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-015701\""
Oct 31 12:47:36.429379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.405671    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc43894-bca9-4020-a8a6-923731346094-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-015701-b497b4869-jfkcx\" (UID: \"3dc43894-bca9-4020-a8a6-923731346094\") " pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"
Oct 31 12:47:36.429379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.405809    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqkhs\" (UniqueName: \"kubernetes.io/projected/3dc43894-bca9-4020-a8a6-923731346094-kube-api-access-cqkhs\") pod \"payload-processing-e2e-inf-a-015701-b497b4869-jfkcx\" (UID: \"3dc43894-bca9-4020-a8a6-923731346094\") " pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"
Oct 31 12:47:36.533498 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.516245    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc43894-bca9-4020-a8a6-923731346094-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-015701-b497b4869-jfkcx\" (UID: \"3dc43894-bca9-4020-a8a6-923731346094\") " pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"
Oct 31 12:47:36.533836 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.526044    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc43894-bca9-4020-a8a6-923731346094-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-015701-b497b4869-jfkcx\" (UID: \"3dc43894-bca9-4020-a8a6-923731346094\") " pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"
Oct 31 12:47:36.534410 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.534306    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqkhs\" (UniqueName: \"kubernetes.io/projected/3dc43894-bca9-4020-a8a6-923731346094-kube-api-access-cqkhs\") pod \"payload-processing-e2e-inf-a-015701-b497b4869-jfkcx\" (UID: \"3dc43894-bca9-4020-a8a6-923731346094\") " pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"
Oct 31 12:47:36.623429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.621121    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqkhs\" (UniqueName: \"kubernetes.io/projected/3dc43894-bca9-4020-a8a6-923731346094-kube-api-access-cqkhs\") pod \"payload-processing-e2e-inf-a-015701-b497b4869-jfkcx\" (UID: \"3dc43894-bca9-4020-a8a6-923731346094\") " pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"
Oct 31 12:47:36.624153 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.623901    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"
Oct 31 12:47:37.002469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.997736    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg" event={"ID":"aca29716-d6b0-4d20-8488-556d6018aefe","Type":"ContainerStarted","Data":"c0c0ac0b52b5a1e37bb1f00ba780afd9518beb9673560af1a7dede74a21e0899"}
Oct 31 12:47:37.002469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.997826    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg" event={"ID":"aca29716-d6b0-4d20-8488-556d6018aefe","Type":"ContainerStarted","Data":"77c7cbca1bfd7560ce61f4c0651a631d792f4a3f8aab89063b7bd47e85fe467a"}
Oct 31 12:47:37.002469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:36.999171    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"
Oct 31 12:47:37.062746 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:37.062654    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg" podStartSLOduration=2.062633067 podStartE2EDuration="2.062633067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:37.044896155 +0000 UTC m=+1181.785322372" watchObservedRunningTime="2026-10-31 12:47:37.062633067 +0000 UTC m=+1181.803059284"
Oct 31 12:47:37.102125 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:37.102097    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"]
Oct 31 12:47:37.104996 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:47:37.104953    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc43894_bca9_4020_a8a6_923731346094.slice/crio-ff957a85d997ddb0609d099445a63ac75c77005a0fdedd53d009e828914317bf WatchSource:0}: Error finding container ff957a85d997ddb0609d099445a63ac75c77005a0fdedd53d009e828914317bf: Status 404 returned error can't find the container with id ff957a85d997ddb0609d099445a63ac75c77005a0fdedd53d009e828914317bf
Oct 31 12:47:38.026094 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:38.026053    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx" event={"ID":"3dc43894-bca9-4020-a8a6-923731346094","Type":"ContainerStarted","Data":"fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807"}
Oct 31 12:47:38.028622 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:38.028594    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"
Oct 31 12:47:38.028832 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:38.028815    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx" event={"ID":"3dc43894-bca9-4020-a8a6-923731346094","Type":"ContainerStarted","Data":"ff957a85d997ddb0609d099445a63ac75c77005a0fdedd53d009e828914317bf"}
Oct 31 12:47:38.068379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:38.060892    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx" podStartSLOduration=2.060872501 podStartE2EDuration="2.060872501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:38.057159494 +0000 UTC m=+1182.797585714" watchObservedRunningTime="2026-10-31 12:47:38.060872501 +0000 UTC m=+1182.801298716"
Oct 31 12:47:41.295688 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:41.295648    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj" podUID="af38a90f-5827-452c-b245-7e147b5983eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:47:42.617657 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:42.617617    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs" podUID="89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:47:43.128975 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.128917    5663 generic.go:356] "Generic (PLEG): container finished" podID="af38a90f-5827-452c-b245-7e147b5983eb" containerID="ec1e82dc1906eeb6f65112e520bd1cb60952c9f883ce448e8068b3d2609faf1b" exitCode=0
Oct 31 12:47:43.128975 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.128970    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj" event={"ID":"af38a90f-5827-452c-b245-7e147b5983eb","Type":"ContainerDied","Data":"ec1e82dc1906eeb6f65112e520bd1cb60952c9f883ce448e8068b3d2609faf1b"}
Oct 31 12:47:43.141175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.141047    5663 generic.go:356] "Generic (PLEG): container finished" podID="89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1" containerID="c4eae499a7bbbbc0b7c7031790d0ae8511d5027ca69ac989118b000b8c39d53f" exitCode=0
Oct 31 12:47:43.141175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.141101    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs" event={"ID":"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1","Type":"ContainerDied","Data":"c4eae499a7bbbbc0b7c7031790d0ae8511d5027ca69ac989118b000b8c39d53f"}
Oct 31 12:47:43.308927 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.308895    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"
Oct 31 12:47:43.465452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.461426    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af38a90f-5827-452c-b245-7e147b5983eb-kube-api-access-b86wk\" (UniqueName: \"kubernetes.io/projected/af38a90f-5827-452c-b245-7e147b5983eb-kube-api-access-b86wk\") pod \"af38a90f-5827-452c-b245-7e147b5983eb\" (UID: \"af38a90f-5827-452c-b245-7e147b5983eb\") "
Oct 31 12:47:43.465452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.461492    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af38a90f-5827-452c-b245-7e147b5983eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af38a90f-5827-452c-b245-7e147b5983eb-plugins-config-volume\") pod \"af38a90f-5827-452c-b245-7e147b5983eb\" (UID: \"af38a90f-5827-452c-b245-7e147b5983eb\") "
Oct 31 12:47:43.499425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.492682    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af38a90f-5827-452c-b245-7e147b5983eb-plugins-config-volume" pod "af38a90f-5827-452c-b245-7e147b5983eb" (UID: "af38a90f-5827-452c-b245-7e147b5983eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:47:43.499425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.495550    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"
Oct 31 12:47:43.504095 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.503882    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af38a90f-5827-452c-b245-7e147b5983eb-kube-api-access-b86wk" pod "af38a90f-5827-452c-b245-7e147b5983eb" (UID: "af38a90f-5827-452c-b245-7e147b5983eb"). InnerVolumeSpecName "kube-api-access-b86wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:43.569245 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.569211    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-plugins-config-volume\") pod \"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1\" (UID: \"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1\") "
Oct 31 12:47:43.586138 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.586106    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-kube-api-access-mpjds\" (UniqueName: \"kubernetes.io/projected/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-kube-api-access-mpjds\") pod \"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1\" (UID: \"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1\") "
Oct 31 12:47:43.587752 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.587728    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b86wk\" (UniqueName: \"kubernetes.io/projected/af38a90f-5827-452c-b245-7e147b5983eb-kube-api-access-b86wk\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:43.605055 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.605031    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af38a90f-5827-452c-b245-7e147b5983eb-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:43.605254 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.585853    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-plugins-config-volume" pod "89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1" (UID: "89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:47:43.618744 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.615981    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-kube-api-access-mpjds" pod "89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1" (UID: "89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1"). InnerVolumeSpecName "kube-api-access-mpjds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:43.718741 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.718676    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:43.718741 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:43.718709    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpjds\" (UniqueName: \"kubernetes.io/projected/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1-kube-api-access-mpjds\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:44.174406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.168617    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs" event={"ID":"89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1","Type":"ContainerDied","Data":"db9a4184bb852492ea469a2ab615b5979784c95b6eb36e4a1ae5cb785d3ab0f8"}
Oct 31 12:47:44.174406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.168697    5663 scope.go:122] "RemoveContainer" containerID="c4eae499a7bbbbc0b7c7031790d0ae8511d5027ca69ac989118b000b8c39d53f"
Oct 31 12:47:44.174406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.168867    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"
Oct 31 12:47:44.321175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.321147    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj" event={"ID":"af38a90f-5827-452c-b245-7e147b5983eb","Type":"ContainerDied","Data":"5a3dac001951b15ca13799acd297e9f6de27d2fe09da8693d38c3702f1c54364"}
Oct 31 12:47:44.321439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.321422    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"
Oct 31 12:47:44.329753 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.329731    5663 scope.go:122] "RemoveContainer" containerID="ec1e82dc1906eeb6f65112e520bd1cb60952c9f883ce448e8068b3d2609faf1b"
Oct 31 12:47:44.333567 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.333539    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"]
Oct 31 12:47:44.337892 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.337868    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-sdfqs"]
Oct 31 12:47:44.351402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.351375    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"]
Oct 31 12:47:44.407459 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.407427    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-rdzwj"]
Oct 31 12:47:44.415503 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.415469    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"]
Oct 31 12:47:44.416156 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.415731    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q" podUID="06231b8b-0712-4804-a2c2-aa0c3f8bee42" containerName="maas-api" containerID="cri-o://483823db1f8a79e265f518f0866f57475b33e1173cd01667f3b4ba89fb9a2b19" gracePeriod=30
Oct 31 12:47:44.761066 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:44.760634    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"
Oct 31 12:47:45.628528 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.628497    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:47:45.642380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.639050    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06231b8b-0712-4804-a2c2-aa0c3f8bee42-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06231b8b-0712-4804-a2c2-aa0c3f8bee42-openshift-service-ca\") pod \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") "
Oct 31 12:47:45.642380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.640226    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1" path="/var/lib/kubelet/pods/89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1/volumes"
Oct 31 12:47:45.642380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.640903    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af38a90f-5827-452c-b245-7e147b5983eb" path="/var/lib/kubelet/pods/af38a90f-5827-452c-b245-7e147b5983eb/volumes"
Oct 31 12:47:45.642380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.641515    5663 generic.go:356] "Generic (PLEG): container finished" podID="06231b8b-0712-4804-a2c2-aa0c3f8bee42" containerID="483823db1f8a79e265f518f0866f57475b33e1173cd01667f3b4ba89fb9a2b19" exitCode=0
Oct 31 12:47:45.642380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.641590    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"
Oct 31 12:47:45.642380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.641714    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q" event={"ID":"06231b8b-0712-4804-a2c2-aa0c3f8bee42","Type":"ContainerDied","Data":"483823db1f8a79e265f518f0866f57475b33e1173cd01667f3b4ba89fb9a2b19"}
Oct 31 12:47:45.642380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.641753    5663 scope.go:122] "RemoveContainer" containerID="483823db1f8a79e265f518f0866f57475b33e1173cd01667f3b4ba89fb9a2b19"
Oct 31 12:47:45.642876 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.642741    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06231b8b-0712-4804-a2c2-aa0c3f8bee42-openshift-service-ca" pod "06231b8b-0712-4804-a2c2-aa0c3f8bee42" (UID: "06231b8b-0712-4804-a2c2-aa0c3f8bee42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:47:45.645723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.645697    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-metrics-certs\") pod \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") "
Oct 31 12:47:45.645842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.645783    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06231b8b-0712-4804-a2c2-aa0c3f8bee42-kube-api-access-mkctn\" (UniqueName: \"kubernetes.io/projected/06231b8b-0712-4804-a2c2-aa0c3f8bee42-kube-api-access-mkctn\") pod \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") "
Oct 31 12:47:45.646973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.646949    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06231b8b-0712-4804-a2c2-aa0c3f8bee42-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:45.701066 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.691881    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-metrics-certs" pod "06231b8b-0712-4804-a2c2-aa0c3f8bee42" (UID: "06231b8b-0712-4804-a2c2-aa0c3f8bee42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:47:45.701066 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.697285    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06231b8b-0712-4804-a2c2-aa0c3f8bee42-kube-api-access-mkctn" pod "06231b8b-0712-4804-a2c2-aa0c3f8bee42" (UID: "06231b8b-0712-4804-a2c2-aa0c3f8bee42"). InnerVolumeSpecName "kube-api-access-mkctn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:47:45.748719 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.748692    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-maas-api-tls\") pod \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\" (UID: \"06231b8b-0712-4804-a2c2-aa0c3f8bee42\") "
Oct 31 12:47:45.749465 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.749447    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:45.749563 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.749554    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkctn\" (UniqueName: \"kubernetes.io/projected/06231b8b-0712-4804-a2c2-aa0c3f8bee42-kube-api-access-mkctn\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:45.751590 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.751570    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-maas-api-tls" pod "06231b8b-0712-4804-a2c2-aa0c3f8bee42" (UID: "06231b8b-0712-4804-a2c2-aa0c3f8bee42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:47:45.895473 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:45.890558    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06231b8b-0712-4804-a2c2-aa0c3f8bee42-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:47:46.453870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:46.453789    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"]
Oct 31 12:47:46.593936 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:46.592599    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f22e7f-65bcd7bb57-7gp4q"]
Oct 31 12:47:47.486792 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:47.486743    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06231b8b-0712-4804-a2c2-aa0c3f8bee42" path="/var/lib/kubelet/pods/06231b8b-0712-4804-a2c2-aa0c3f8bee42/volumes"
Oct 31 12:47:49.034669 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:49.034638    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"
Oct 31 12:47:49.035156 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:49.034722    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"
Oct 31 12:47:50.325525 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:50.325476    5663 patch_prober.go:29] interesting pod/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:47:50.326169 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:50.325562    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:47:52.480022 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:47:52.479982    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:47:53.890019 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.889980    5663 generic.go:356] "Generic (PLEG): container finished" podID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerID="118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d" exitCode=0
Oct 31 12:47:53.890517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.890043    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" event={"ID":"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0","Type":"ContainerDied","Data":"118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d"}
Oct 31 12:47:53.890629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.890612    5663 scope.go:122] "RemoveContainer" containerID="118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d"
Oct 31 12:47:53.892654 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.892622    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:47:53.892747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.892693    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:47:53.892805 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.892753    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:47:53.981951 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.981921    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"]
Oct 31 12:47:53.982668 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.982653    5663 cpu_manager.go:414] "containerMap: removing container" podUID="89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1" containerName="payload-processing"
Oct 31 12:47:53.982735 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.982671    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1" containerName="payload-processing"
Oct 31 12:47:53.982772 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.982739    5663 cpu_manager.go:414] "containerMap: removing container" podUID="af38a90f-5827-452c-b245-7e147b5983eb" containerName="payload-pre-processing"
Oct 31 12:47:53.982772 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.982748    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af38a90f-5827-452c-b245-7e147b5983eb" containerName="payload-pre-processing"
Oct 31 12:47:53.982772 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.982759    5663 cpu_manager.go:414] "containerMap: removing container" podUID="06231b8b-0712-4804-a2c2-aa0c3f8bee42" containerName="maas-api"
Oct 31 12:47:53.982772 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.982767    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06231b8b-0712-4804-a2c2-aa0c3f8bee42" containerName="maas-api"
Oct 31 12:47:53.982927 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.982915    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="af38a90f-5827-452c-b245-7e147b5983eb" containerName="payload-pre-processing"
Oct 31 12:47:53.982964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.982928    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="06231b8b-0712-4804-a2c2-aa0c3f8bee42" containerName="maas-api"
Oct 31 12:47:53.982964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.982941    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d56a73-8e70-4d00-8c7f-fb5ae35ed3c1" containerName="payload-processing"
Oct 31 12:47:53.985993 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:53.985973    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"
Oct 31 12:47:54.003702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.003453    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e1fc0ffe\""
Oct 31 12:47:54.034103 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.034065    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"]
Oct 31 12:47:54.066136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.066055    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"]
Oct 31 12:47:54.071667 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.071637    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"
Oct 31 12:47:54.094035 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.094005    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e1fc0ffe-dockercfg-x4r9q\""
Oct 31 12:47:54.098242 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.098214    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e307b76-8afc-4b60-9996-a980104474ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p\" (UID: \"1e307b76-8afc-4b60-9996-a980104474ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"
Oct 31 12:47:54.098422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.098311    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drgj4\" (UniqueName: \"kubernetes.io/projected/1e307b76-8afc-4b60-9996-a980104474ef-kube-api-access-drgj4\") pod \"payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p\" (UID: \"1e307b76-8afc-4b60-9996-a980104474ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"
Oct 31 12:47:54.112590 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.112551    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"]
Oct 31 12:47:54.215423 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.200108    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzllv\" (UniqueName: \"kubernetes.io/projected/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-kube-api-access-pzllv\") pod \"payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh\" (UID: \"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a\") " pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"
Oct 31 12:47:54.215423 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.200515    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e307b76-8afc-4b60-9996-a980104474ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p\" (UID: \"1e307b76-8afc-4b60-9996-a980104474ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"
Oct 31 12:47:54.215423 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.200621    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drgj4\" (UniqueName: \"kubernetes.io/projected/1e307b76-8afc-4b60-9996-a980104474ef-kube-api-access-drgj4\") pod \"payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p\" (UID: \"1e307b76-8afc-4b60-9996-a980104474ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"
Oct 31 12:47:54.215423 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.200661    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh\" (UID: \"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a\") " pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"
Oct 31 12:47:54.215423 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.202001    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e307b76-8afc-4b60-9996-a980104474ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p\" (UID: \"1e307b76-8afc-4b60-9996-a980104474ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"
Oct 31 12:47:54.222381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.218855    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drgj4\" (UniqueName: \"kubernetes.io/projected/1e307b76-8afc-4b60-9996-a980104474ef-kube-api-access-drgj4\") pod \"payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p\" (UID: \"1e307b76-8afc-4b60-9996-a980104474ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"
Oct 31 12:47:54.309378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.308937    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh\" (UID: \"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a\") " pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"
Oct 31 12:47:54.309378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.309004    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzllv\" (UniqueName: \"kubernetes.io/projected/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-kube-api-access-pzllv\") pod \"payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh\" (UID: \"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a\") " pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"
Oct 31 12:47:54.313619 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.312937    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"
Oct 31 12:47:54.321376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.315238    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh\" (UID: \"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a\") " pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"
Oct 31 12:47:54.383528 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.381917    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzllv\" (UniqueName: \"kubernetes.io/projected/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-kube-api-access-pzllv\") pod \"payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh\" (UID: \"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a\") " pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"
Oct 31 12:47:54.397372 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:54.396714    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"
Oct 31 12:47:55.077555 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:55.077484    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" event={"ID":"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0","Type":"ContainerStarted","Data":"56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838"}
Oct 31 12:47:55.349377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:55.345116    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"]
Oct 31 12:47:55.636377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:55.602466    5663 patch_prober.go:29] interesting pod/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 12:47:55.636377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:55.602528    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 12:47:55.636377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:55.609626    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:47:55.636377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:55.609698    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"]
Oct 31 12:47:56.089192 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:56.089151    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p" event={"ID":"1e307b76-8afc-4b60-9996-a980104474ef","Type":"ContainerStarted","Data":"8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc"}
Oct 31 12:47:56.089627 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:56.089220    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p" event={"ID":"1e307b76-8afc-4b60-9996-a980104474ef","Type":"ContainerStarted","Data":"0dfd989e31bf303d6a1bf3689144a8476ca08dacf6e1a2c490a82de187116095"}
Oct 31 12:47:56.089627 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:56.089252    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"
Oct 31 12:47:56.090939 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:56.090911    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh" event={"ID":"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a","Type":"ContainerStarted","Data":"8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e"}
Oct 31 12:47:56.091062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:56.090965    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh" event={"ID":"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a","Type":"ContainerStarted","Data":"d76a11f0b1797291758f3fcc3b97434874514f58e869728266830da84f3327f2"}
Oct 31 12:47:56.091137 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:56.091073    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"
Oct 31 12:47:56.114465 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:56.114413    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p" podStartSLOduration=3.114399716 podStartE2EDuration="3.114399716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:56.114162811 +0000 UTC m=+1200.854589038" watchObservedRunningTime="2026-10-31 12:47:56.114399716 +0000 UTC m=+1200.854825938"
Oct 31 12:47:56.145458 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:56.145409    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh" podStartSLOduration=2.145394361 podStartE2EDuration="2.145394361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:47:56.142871443 +0000 UTC m=+1200.883297683" watchObservedRunningTime="2026-10-31 12:47:56.145394361 +0000 UTC m=+1200.885820578"
Oct 31 12:47:56.593221 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:56.593180    5663 patch_prober.go:29] interesting pod/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 12:47:56.593418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:56.593239    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 12:47:57.594376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:57.593937    5663 patch_prober.go:29] interesting pod/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 12:47:57.594376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:57.594045    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 12:47:58.602432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:58.598967    5663 patch_prober.go:29] interesting pod/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 12:47:58.602432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:58.599031    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 12:47:59.592907 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:59.592872    5663 patch_prober.go:29] interesting pod/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 12:47:59.593125 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:47:59.592924    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 12:48:00.605446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:00.598865    5663 patch_prober.go:29] interesting pod/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 12:48:00.605446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:00.598934    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 12:48:01.592757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:01.592721    5663 patch_prober.go:29] interesting pod/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 31 12:48:01.592941 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:01.592773    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 31 12:48:02.597618 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:02.597586    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:48:02.598076 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:02.598053    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:48:02.598690 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:02.598672    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:48:07.101672 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:07.101641    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"
Oct 31 12:48:07.102151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:07.101722    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"
Oct 31 12:48:08.257041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.257004    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"]
Oct 31 12:48:08.261306 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.261263    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.266274 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.266252    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-fb2b1d-openshift-default-dockercfg-6b2h6\""
Oct 31 12:48:08.283029 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.282998    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"]
Oct 31 12:48:08.399009 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.398973    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.399167 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.399088    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.399167 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.399158    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.399262 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.399223    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.399325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.399310    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.399480 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.399457    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xll87\" (UniqueName: \"kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-kube-api-access-xll87\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.399542 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.399529    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.399635 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.399621    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.399685 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.399673    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.500877 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.500835    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xll87\" (UniqueName: \"kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-kube-api-access-xll87\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.501079 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.500924    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.501079 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.501023    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.501211 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.501083    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.501211 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.501157    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.501331 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.501264    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.501431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.501407    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.501617 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.501596    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.501801 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.501759    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-certs\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.501958 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.501650    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.502084 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.501808    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.502208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.501884    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-credential-socket\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.502208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.501943    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.502208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.502063    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-data\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.503616 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.503594    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-envoy\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.503769 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.503751    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-podinfo\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.513612 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.513555    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-token\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.513773 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.513751    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xll87\" (UniqueName: \"kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-kube-api-access-xll87\") pod \"e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") " pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.575384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.575333    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:08.907435 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.905618    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"]
Oct 31 12:48:08.968501 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.961945    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:08.968501 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.962052    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:08.968501 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:08.962095    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:09.189443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:09.189343    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" event={"ID":"5855c115-9e7a-46b0-89e8-d7ec52b1719b","Type":"ContainerStarted","Data":"d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc"}
Oct 31 12:48:09.189443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:09.189408    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" event={"ID":"5855c115-9e7a-46b0-89e8-d7ec52b1719b","Type":"ContainerStarted","Data":"19c6a1457a10a682025a254b2ca3d1528dc87ac6c4294339719a11fb18db9adf"}
Oct 31 12:48:09.217624 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:09.217564    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" podStartSLOduration=1.2175433629999999 podStartE2EDuration="1.217543363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:09.215346135 +0000 UTC m=+1213.955772351" watchObservedRunningTime="2026-10-31 12:48:09.217543363 +0000 UTC m=+1213.957969580"
Oct 31 12:48:09.582595 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:09.582555    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:10.589551 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:10.589511    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 12:48:10.590234 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:10.590190    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" podUID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 12:48:11.582933 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:11.582894    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 12:48:11.583212 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:11.583178    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" podUID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 12:48:12.584481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:12.575796    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 12:48:12.584481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:12.575858    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" podUID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 12:48:13.582971 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:13.582934    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 12:48:13.583270 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:13.583239    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" podUID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 12:48:14.580383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:14.579822    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 12:48:14.580383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:14.579885    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" podUID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 12:48:15.577374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:15.576015    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 12:48:15.577374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:15.576078    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" podUID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 12:48:15.972377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:15.969173    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"]
Oct 31 12:48:16.039388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.030974    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.311428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.279840    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-token\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.311428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.279936    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-envoy\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.311428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.280001    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-credential-socket\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.311428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.280165    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkms\" (UniqueName: \"kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-kube-api-access-jvkms\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.311428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.280225    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-certs\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.311428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.280249    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-podinfo\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.311428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.280303    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istiod-ca-cert\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.311428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.280397    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-data\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.311428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.280546    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-socket\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.311428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.305909    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ad1af7-openshift-default-dockercfg-9zj6x\""
Oct 31 12:48:16.462231 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.462194    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-credential-socket\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.462585 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.462564    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkms\" (UniqueName: \"kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-kube-api-access-jvkms\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.462725 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.462712    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-certs\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.462826 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.462812    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-podinfo\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.462944 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.462929    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istiod-ca-cert\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.463063 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.463050    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-data\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.463251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.463238    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-socket\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.465441 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.465414    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-certs\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.465786 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.465766    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"]
Oct 31 12:48:16.485130 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.485096    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-credential-socket\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.486073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.486044    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-data\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.489383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.487076    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istiod-ca-cert\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.497958 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.497927    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-socket\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.499547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.499514    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-token\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.499793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.499772    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-envoy\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.526002 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.525966    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-token\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.603606 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.602928    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:48:16.603606 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.603585    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 31 12:48:16.603995 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.603965    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:48:16.608374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.605041    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkms\" (UniqueName: \"kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-kube-api-access-jvkms\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.640600 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.627720    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:16.640600 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.639034    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:48:16.640600 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.639292    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:48:16.726652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.719177    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 12:48:16.729742 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.727059    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" podUID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 12:48:16.810085 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.810044    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-podinfo\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:16.841505 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:16.840779    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-envoy\") pod \"e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") " pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:17.151289 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:17.151253    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:17.562571 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:17.560459    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:17.562571 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:17.560554    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:17.562571 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:17.560600    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:17.573329 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:17.572553    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"]
Oct 31 12:48:17.602079 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:17.602053    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:17.603065 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:17.603042    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:17.603891 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:17.603869    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:48:18.512426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:18.512383    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" event={"ID":"9a7ea3d2-2730-4332-8b30-dd2400e2239d","Type":"ContainerStarted","Data":"0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58"}
Oct 31 12:48:18.512840 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:18.512446    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" event={"ID":"9a7ea3d2-2730-4332-8b30-dd2400e2239d","Type":"ContainerStarted","Data":"2b11fff71ed79abbdc50927191e0b88117895c4b133fb05f48688a2b179531f9"}
Oct 31 12:48:18.547134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:18.547077    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" podStartSLOduration=3.547058983 podStartE2EDuration="3.547058983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:18.542116269 +0000 UTC m=+1223.282542497" watchObservedRunningTime="2026-10-31 12:48:18.547058983 +0000 UTC m=+1223.287485200"
Oct 31 12:48:19.152322 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:19.152289    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:19.158392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:19.158340    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:19.520945 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:19.520846    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:19.521892 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:19.521867    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:48:22.479522 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:22.479487    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:48:31.559059 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:31.559024    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7029-b44bc1"]
Oct 31 12:48:31.566497 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:31.566468    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-b44bc1"
Oct 31 12:48:31.573685 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:31.573660    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-klvx2\""
Oct 31 12:48:31.575788 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:31.575761    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7029-b44bc1"]
Oct 31 12:48:31.687638 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:31.665978    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4k6h\" (UniqueName: \"kubernetes.io/projected/307d5358-5c91-4a06-a9d3-e9af2dac43bb-kube-api-access-s4k6h\") pod \"test-curl-7029-b44bc1\" (UID: \"307d5358-5c91-4a06-a9d3-e9af2dac43bb\") " pod="opendatahub/test-curl-7029-b44bc1"
Oct 31 12:48:31.840431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:31.801110    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4k6h\" (UniqueName: \"kubernetes.io/projected/307d5358-5c91-4a06-a9d3-e9af2dac43bb-kube-api-access-s4k6h\") pod \"test-curl-7029-b44bc1\" (UID: \"307d5358-5c91-4a06-a9d3-e9af2dac43bb\") " pod="opendatahub/test-curl-7029-b44bc1"
Oct 31 12:48:32.072977 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:32.072940    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4k6h\" (UniqueName: \"kubernetes.io/projected/307d5358-5c91-4a06-a9d3-e9af2dac43bb-kube-api-access-s4k6h\") pod \"test-curl-7029-b44bc1\" (UID: \"307d5358-5c91-4a06-a9d3-e9af2dac43bb\") " pod="opendatahub/test-curl-7029-b44bc1"
Oct 31 12:48:32.194196 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:32.194114    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-b44bc1"
Oct 31 12:48:32.618672 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:32.611957    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7029-b44bc1"]
Oct 31 12:48:33.585339 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:33.585305    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"]
Oct 31 12:48:33.587854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:33.587831    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"]
Oct 31 12:48:33.588242 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:33.588130    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7029-b44bc1" event={"ID":"307d5358-5c91-4a06-a9d3-e9af2dac43bb","Type":"ContainerStarted","Data":"8de1ad7669a22b88f2561bf1db1e302e0eb0db86521a42d2019ba33782a81686"}
Oct 31 12:48:33.588959 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:33.588930    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9" podUID="6f0a7567-dc4e-495b-a718-1bc2ca817635" containerName="maas-api" containerID="cri-o://39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182" gracePeriod=30
Oct 31 12:48:33.615677 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:33.615651    5663 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-api-a-5c5815: configmap "payload-processing-plugins-e2e-api-a-5c5815" not found
Oct 31 12:48:33.615871 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:33.615735    5663 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume podName:119e2bed-7181-48bc-a8f2-8be78a229cee nodeName:}" failed. No retries permitted until 2026-10-31 12:48:34.115714794 +0000 UTC m=+1238.856141006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume") pod "payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd" (UID: "119e2bed-7181-48bc-a8f2-8be78a229cee") : configmap "payload-processing-plugins-e2e-api-a-5c5815" not found
Oct 31 12:48:34.127298 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:34.127221    5663 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-api-a-5c5815: configmap "payload-processing-plugins-e2e-api-a-5c5815" not found
Oct 31 12:48:34.127950 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:34.127930    5663 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume podName:119e2bed-7181-48bc-a8f2-8be78a229cee nodeName:}" failed. No retries permitted until 2026-10-31 12:48:35.127906268 +0000 UTC m=+1239.868332481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume") pod "payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd" (UID: "119e2bed-7181-48bc-a8f2-8be78a229cee") : configmap "payload-processing-plugins-e2e-api-a-5c5815" not found
Oct 31 12:48:34.295327 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.295294    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:48:34.341522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.341396    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-metrics-certs\") pod \"6f0a7567-dc4e-495b-a718-1bc2ca817635\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") "
Oct 31 12:48:34.341716 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.341528    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-maas-api-tls\") pod \"6f0a7567-dc4e-495b-a718-1bc2ca817635\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") "
Oct 31 12:48:34.341716 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.341685    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0a7567-dc4e-495b-a718-1bc2ca817635-kube-api-access-7676g\" (UniqueName: \"kubernetes.io/projected/6f0a7567-dc4e-495b-a718-1bc2ca817635-kube-api-access-7676g\") pod \"6f0a7567-dc4e-495b-a718-1bc2ca817635\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") "
Oct 31 12:48:34.341848 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.341830    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0a7567-dc4e-495b-a718-1bc2ca817635-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0a7567-dc4e-495b-a718-1bc2ca817635-openshift-service-ca\") pod \"6f0a7567-dc4e-495b-a718-1bc2ca817635\" (UID: \"6f0a7567-dc4e-495b-a718-1bc2ca817635\") "
Oct 31 12:48:34.345181 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.343083    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0a7567-dc4e-495b-a718-1bc2ca817635-openshift-service-ca" pod "6f0a7567-dc4e-495b-a718-1bc2ca817635" (UID: "6f0a7567-dc4e-495b-a718-1bc2ca817635"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:48:34.374716 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.374680    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-maas-api-tls" pod "6f0a7567-dc4e-495b-a718-1bc2ca817635" (UID: "6f0a7567-dc4e-495b-a718-1bc2ca817635"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:48:34.378624 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.378551    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-metrics-certs" pod "6f0a7567-dc4e-495b-a718-1bc2ca817635" (UID: "6f0a7567-dc4e-495b-a718-1bc2ca817635"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:48:34.385687 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.383261    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0a7567-dc4e-495b-a718-1bc2ca817635-kube-api-access-7676g" pod "6f0a7567-dc4e-495b-a718-1bc2ca817635" (UID: "6f0a7567-dc4e-495b-a718-1bc2ca817635"). InnerVolumeSpecName "kube-api-access-7676g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:34.444803 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.444765    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7676g\" (UniqueName: \"kubernetes.io/projected/6f0a7567-dc4e-495b-a718-1bc2ca817635-kube-api-access-7676g\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:34.444803 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.444798    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0a7567-dc4e-495b-a718-1bc2ca817635-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:34.444803 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.444811    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:34.445117 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.444823    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f0a7567-dc4e-495b-a718-1bc2ca817635-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:34.576175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.575703    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7029-b44bc1" event={"ID":"307d5358-5c91-4a06-a9d3-e9af2dac43bb","Type":"ContainerStarted","Data":"76985b2427004218468a818123a0ea0ca9beac9523b87da3ba9d0cf954ac1600"}
Oct 31 12:48:34.579424 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.579390    5663 generic.go:356] "Generic (PLEG): container finished" podID="6f0a7567-dc4e-495b-a718-1bc2ca817635" containerID="39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182" exitCode=0
Oct 31 12:48:34.580152 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.580130    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"
Oct 31 12:48:34.589393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.583667    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9" event={"ID":"6f0a7567-dc4e-495b-a718-1bc2ca817635","Type":"ContainerDied","Data":"39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182"}
Oct 31 12:48:34.589393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.583748    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9" event={"ID":"6f0a7567-dc4e-495b-a718-1bc2ca817635","Type":"ContainerDied","Data":"0ee5236f4f298b09a2d986ae706b8c6e6515f4286a2be17cb10d899d2bc26650"}
Oct 31 12:48:34.589393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.583791    5663 scope.go:122] "RemoveContainer" containerID="39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182"
Oct 31 12:48:34.616756 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.616694    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7029-b44bc1" podStartSLOduration=3.616672758 podStartE2EDuration="3.616672758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:34.605204527 +0000 UTC m=+1239.345630744" watchObservedRunningTime="2026-10-31 12:48:34.616672758 +0000 UTC m=+1239.357098974"
Oct 31 12:48:34.688246 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.686518    5663 scope.go:122] "RemoveContainer" containerID="39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182"
Oct 31 12:48:34.698995 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:34.695643    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182\": container with ID starting with 39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182 not found: ID does not exist" containerID="39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182"
Oct 31 12:48:34.698995 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.695691    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182"} err="failed to get container status \"39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182\": rpc error: code = NotFound desc = could not find container \"39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182\": container with ID starting with 39c2b09ef16a6f302658ff9723e9ca31592111e2deea435dfe5a6630f3577182 not found: ID does not exist"
Oct 31 12:48:34.727559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.727521    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"]
Oct 31 12:48:34.732854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:34.732805    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bb3cf5-7fd658679c-rzdb9"]
Oct 31 12:48:35.204295 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:35.204261    5663 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-api-a-5c5815: configmap "payload-processing-plugins-e2e-api-a-5c5815" not found
Oct 31 12:48:35.204731 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:35.204365    5663 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume podName:119e2bed-7181-48bc-a8f2-8be78a229cee nodeName:}" failed. No retries permitted until 2026-10-31 12:48:37.204330259 +0000 UTC m=+1241.944756471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume") pod "payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd" (UID: "119e2bed-7181-48bc-a8f2-8be78a229cee") : configmap "payload-processing-plugins-e2e-api-a-5c5815" not found
Oct 31 12:48:35.747982 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:35.747946    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0a7567-dc4e-495b-a718-1bc2ca817635" path="/var/lib/kubelet/pods/6f0a7567-dc4e-495b-a718-1bc2ca817635/volumes"
Oct 31 12:48:35.982389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:35.982344    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7029-b44bc1"]
Oct 31 12:48:35.994486 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:35.994455    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7029-b44bc1"]
Oct 31 12:48:36.819378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:36.818690    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7029-b44bc1" podUID="307d5358-5c91-4a06-a9d3-e9af2dac43bb" containerName="test-curl-7029-b44bc1" containerID="cri-o://76985b2427004218468a818123a0ea0ca9beac9523b87da3ba9d0cf954ac1600" gracePeriod=2
Oct 31 12:48:36.824607 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:36.824564    5663 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7029-b44bc1\" is forbidden: User \"system:node:ip-10-0-140-137.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-137.us-east-2.compute.internal' and this object" podUID="307d5358-5c91-4a06-a9d3-e9af2dac43bb" pod="opendatahub/test-curl-7029-b44bc1"
Oct 31 12:48:37.245437 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:37.240896    5663 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-api-a-5c5815: configmap "payload-processing-plugins-e2e-api-a-5c5815" not found
Oct 31 12:48:37.245437 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:37.240988    5663 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume podName:119e2bed-7181-48bc-a8f2-8be78a229cee nodeName:}" failed. No retries permitted until 2026-10-31 12:48:41.240965447 +0000 UTC m=+1245.981391645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume") pod "payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd" (UID: "119e2bed-7181-48bc-a8f2-8be78a229cee") : configmap "payload-processing-plugins-e2e-api-a-5c5815" not found
Oct 31 12:48:38.873163 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:38.871187    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:48:38.895491 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:38.878762    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:39.055461 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.055413    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:39.162463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.115490    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-ms7fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:48:39.162463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.115559    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:39.342381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.300537    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"]
Oct 31 12:48:39.342381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.301778    5663 cpu_manager.go:414] "containerMap: removing container" podUID="307d5358-5c91-4a06-a9d3-e9af2dac43bb" containerName="test-curl-7029-b44bc1"
Oct 31 12:48:39.342381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.301802    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307d5358-5c91-4a06-a9d3-e9af2dac43bb" containerName="test-curl-7029-b44bc1"
Oct 31 12:48:39.342381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.301863    5663 cpu_manager.go:414] "containerMap: removing container" podUID="6f0a7567-dc4e-495b-a718-1bc2ca817635" containerName="maas-api"
Oct 31 12:48:39.342381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.301871    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0a7567-dc4e-495b-a718-1bc2ca817635" containerName="maas-api"
Oct 31 12:48:39.342381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.302207    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0a7567-dc4e-495b-a718-1bc2ca817635" containerName="maas-api"
Oct 31 12:48:39.342381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.302236    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="307d5358-5c91-4a06-a9d3-e9af2dac43bb" containerName="test-curl-7029-b44bc1"
Oct 31 12:48:39.529690 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.529598    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"]
Oct 31 12:48:39.595481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.595424    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd" podUID="119e2bed-7181-48bc-a8f2-8be78a229cee" containerName="payload-pre-processing" containerID="cri-o://aae2c5e84066418a6043a50d9cd7186e27e120ecf7037bb4c00cebcfc9357530" gracePeriod=25
Oct 31 12:48:39.595967 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.595842    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:48:39.596758 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.596736    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:48:39.691506 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.683506    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w" podUID="ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:39.719514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.687950    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"
Oct 31 12:48:39.719514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.712077    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4e8eb9\""
Oct 31 12:48:39.749380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.720478    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:48:39.749380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.722166    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c1faf-d346-4596-9e50-ab782ae40b2f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l\" (UID: \"1c8c1faf-d346-4596-9e50-ab782ae40b2f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"
Oct 31 12:48:39.749380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.722230    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2hf\" (UniqueName: \"kubernetes.io/projected/1c8c1faf-d346-4596-9e50-ab782ae40b2f-kube-api-access-bz2hf\") pod \"payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l\" (UID: \"1c8c1faf-d346-4596-9e50-ab782ae40b2f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"
Oct 31 12:48:39.816825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.794969    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4e8eb9-dockercfg-n9cjm\""
Oct 31 12:48:39.818444 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.818412    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"
Oct 31 12:48:39.875616 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.874098    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c1faf-d346-4596-9e50-ab782ae40b2f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l\" (UID: \"1c8c1faf-d346-4596-9e50-ab782ae40b2f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"
Oct 31 12:48:39.875616 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.874177    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2hf\" (UniqueName: \"kubernetes.io/projected/1c8c1faf-d346-4596-9e50-ab782ae40b2f-kube-api-access-bz2hf\") pod \"payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l\" (UID: \"1c8c1faf-d346-4596-9e50-ab782ae40b2f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"
Oct 31 12:48:39.876203 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.875909    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c1faf-d346-4596-9e50-ab782ae40b2f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l\" (UID: \"1c8c1faf-d346-4596-9e50-ab782ae40b2f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"
Oct 31 12:48:39.902405 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.901578    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:39.920831 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.920467    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:39.985760 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.976010    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xmph\" (UniqueName: \"kubernetes.io/projected/41979d00-c01e-4bfa-953f-f70f7fc4d22b-kube-api-access-5xmph\") pod \"payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn\" (UID: \"41979d00-c01e-4bfa-953f-f70f7fc4d22b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"
Oct 31 12:48:39.985760 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:39.976121    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41979d00-c01e-4bfa-953f-f70f7fc4d22b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn\" (UID: \"41979d00-c01e-4bfa-953f-f70f7fc4d22b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"
Oct 31 12:48:40.036303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.025764    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2hf\" (UniqueName: \"kubernetes.io/projected/1c8c1faf-d346-4596-9e50-ab782ae40b2f-kube-api-access-bz2hf\") pod \"payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l\" (UID: \"1c8c1faf-d346-4596-9e50-ab782ae40b2f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"
Oct 31 12:48:40.043469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.038989    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"]
Oct 31 12:48:40.043469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.039115    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:48:40.043469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.039159    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:48:40.043469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.039177    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"]
Oct 31 12:48:40.118417 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.084293    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"
Oct 31 12:48:40.118417 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.113684    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:40.118417 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.114945    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:48:40.162383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.139291    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xmph\" (UniqueName: \"kubernetes.io/projected/41979d00-c01e-4bfa-953f-f70f7fc4d22b-kube-api-access-5xmph\") pod \"payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn\" (UID: \"41979d00-c01e-4bfa-953f-f70f7fc4d22b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"
Oct 31 12:48:40.211380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.165968    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41979d00-c01e-4bfa-953f-f70f7fc4d22b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn\" (UID: \"41979d00-c01e-4bfa-953f-f70f7fc4d22b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"
Oct 31 12:48:40.212987 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.175002    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41979d00-c01e-4bfa-953f-f70f7fc4d22b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn\" (UID: \"41979d00-c01e-4bfa-953f-f70f7fc4d22b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"
Oct 31 12:48:40.213175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.190478    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:40.346343 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.336959    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 31 12:48:40.468345 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.468260    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xmph\" (UniqueName: \"kubernetes.io/projected/41979d00-c01e-4bfa-953f-f70f7fc4d22b-kube-api-access-5xmph\") pod \"payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn\" (UID: \"41979d00-c01e-4bfa-953f-f70f7fc4d22b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"
Oct 31 12:48:40.473973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.473937    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:48:40.552383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.550415    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:48:40.754517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.732614    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:48:40.754517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.740009    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"
Oct 31 12:48:40.843951 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.843914    5663 generic.go:356] "Generic (PLEG): container finished" podID="307d5358-5c91-4a06-a9d3-e9af2dac43bb" containerID="76985b2427004218468a818123a0ea0ca9beac9523b87da3ba9d0cf954ac1600" exitCode=137
Oct 31 12:48:40.846998 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.846945    5663 generic.go:356] "Generic (PLEG): container finished" podID="119e2bed-7181-48bc-a8f2-8be78a229cee" containerID="aae2c5e84066418a6043a50d9cd7186e27e120ecf7037bb4c00cebcfc9357530" exitCode=0
Oct 31 12:48:40.848856 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.847226    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd" event={"ID":"119e2bed-7181-48bc-a8f2-8be78a229cee","Type":"ContainerDied","Data":"aae2c5e84066418a6043a50d9cd7186e27e120ecf7037bb4c00cebcfc9357530"}
Oct 31 12:48:40.887799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.862011    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-dswld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:48:40.887799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:40.862066    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" podUID="57ad591e-3b45-4312-8612-cf848dcf55e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:41.098044 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.097625    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"
Oct 31 12:48:41.120775 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.120702    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-b44bc1"
Oct 31 12:48:41.147764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.147730    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119e2bed-7181-48bc-a8f2-8be78a229cee-kube-api-access-7dwfm\" (UniqueName: \"kubernetes.io/projected/119e2bed-7181-48bc-a8f2-8be78a229cee-kube-api-access-7dwfm\") pod \"119e2bed-7181-48bc-a8f2-8be78a229cee\" (UID: \"119e2bed-7181-48bc-a8f2-8be78a229cee\") "
Oct 31 12:48:41.147945 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.147867    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume\") pod \"119e2bed-7181-48bc-a8f2-8be78a229cee\" (UID: \"119e2bed-7181-48bc-a8f2-8be78a229cee\") "
Oct 31 12:48:41.148118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.148102    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307d5358-5c91-4a06-a9d3-e9af2dac43bb-kube-api-access-s4k6h\" (UniqueName: \"kubernetes.io/projected/307d5358-5c91-4a06-a9d3-e9af2dac43bb-kube-api-access-s4k6h\") pod \"307d5358-5c91-4a06-a9d3-e9af2dac43bb\" (UID: \"307d5358-5c91-4a06-a9d3-e9af2dac43bb\") "
Oct 31 12:48:41.149506 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.149476    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume" pod "119e2bed-7181-48bc-a8f2-8be78a229cee" (UID: "119e2bed-7181-48bc-a8f2-8be78a229cee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:48:41.154396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.153403    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307d5358-5c91-4a06-a9d3-e9af2dac43bb-kube-api-access-s4k6h" pod "307d5358-5c91-4a06-a9d3-e9af2dac43bb" (UID: "307d5358-5c91-4a06-a9d3-e9af2dac43bb"). InnerVolumeSpecName "kube-api-access-s4k6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:41.154396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.153490    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119e2bed-7181-48bc-a8f2-8be78a229cee-kube-api-access-7dwfm" pod "119e2bed-7181-48bc-a8f2-8be78a229cee" (UID: "119e2bed-7181-48bc-a8f2-8be78a229cee"). InnerVolumeSpecName "kube-api-access-7dwfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:41.155887 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.155817    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"]
Oct 31 12:48:41.251164 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.250953    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/119e2bed-7181-48bc-a8f2-8be78a229cee-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:41.251164 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.250988    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4k6h\" (UniqueName: \"kubernetes.io/projected/307d5358-5c91-4a06-a9d3-e9af2dac43bb-kube-api-access-s4k6h\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:41.251164 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.251004    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dwfm\" (UniqueName: \"kubernetes.io/projected/119e2bed-7181-48bc-a8f2-8be78a229cee-kube-api-access-7dwfm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:41.386790 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.378773    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"]
Oct 31 12:48:41.442527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:41.442503    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307d5358-5c91-4a06-a9d3-e9af2dac43bb" path="/var/lib/kubelet/pods/307d5358-5c91-4a06-a9d3-e9af2dac43bb/volumes"
Oct 31 12:48:42.009426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.009390    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7029-b44bc1"
Oct 31 12:48:42.009947 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.009451    5663 scope.go:122] "RemoveContainer" containerID="76985b2427004218468a818123a0ea0ca9beac9523b87da3ba9d0cf954ac1600"
Oct 31 12:48:42.011484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.011463    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"
Oct 31 12:48:42.011610 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.011527    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd" event={"ID":"119e2bed-7181-48bc-a8f2-8be78a229cee","Type":"ContainerDied","Data":"20d3ee4031adaa7f9c7d8390c84586ab951ca319bc3cb39335a98abc14e8d727"}
Oct 31 12:48:42.014368 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.014333    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn" event={"ID":"41979d00-c01e-4bfa-953f-f70f7fc4d22b","Type":"ContainerStarted","Data":"d6b16fb4e4a6512c1c429b6149bed7897ea6101ad2646e20638e4766b727fbef"}
Oct 31 12:48:42.014525 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.014400    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn" event={"ID":"41979d00-c01e-4bfa-953f-f70f7fc4d22b","Type":"ContainerStarted","Data":"7003b111687d89d483ce33109ce36f5eccbebb4b74e4b61cefbecfeab5ee1e15"}
Oct 31 12:48:42.014525 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.014431    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"
Oct 31 12:48:42.017280 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.016543    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l" event={"ID":"1c8c1faf-d346-4596-9e50-ab782ae40b2f","Type":"ContainerStarted","Data":"ef76890a0d8143f09f02e643b32eb35b37fa8a9ff8b0ce30a3b49cb84fa005ac"}
Oct 31 12:48:42.017280 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.016587    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l" event={"ID":"1c8c1faf-d346-4596-9e50-ab782ae40b2f","Type":"ContainerStarted","Data":"f410b101ce9a6804288058aa39ef2c92e3992a354ebb38aed833bd1085ed0454"}
Oct 31 12:48:42.017280 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.016643    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"
Oct 31 12:48:42.022762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.022742    5663 scope.go:122] "RemoveContainer" containerID="aae2c5e84066418a6043a50d9cd7186e27e120ecf7037bb4c00cebcfc9357530"
Oct 31 12:48:42.038145 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.038104    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn" podStartSLOduration=4.03809147 podStartE2EDuration="4.03809147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:42.036118525 +0000 UTC m=+1246.776544744" watchObservedRunningTime="2026-10-31 12:48:42.03809147 +0000 UTC m=+1246.778517678"
Oct 31 12:48:42.064429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.058078    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"]
Oct 31 12:48:42.064429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:42.064041    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5c5815-7ff5bd68cf-gvdkd"]
Oct 31 12:48:43.440948 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:43.440914    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119e2bed-7181-48bc-a8f2-8be78a229cee" path="/var/lib/kubelet/pods/119e2bed-7181-48bc-a8f2-8be78a229cee/volumes"
Oct 31 12:48:44.977377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:44.968928    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l" podStartSLOduration=6.968903781 podStartE2EDuration="6.968903781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:42.089808431 +0000 UTC m=+1246.830234648" watchObservedRunningTime="2026-10-31 12:48:44.968903781 +0000 UTC m=+1249.709329989"
Oct 31 12:48:45.473374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.435454    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"]
Oct 31 12:48:45.473374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.437699    5663 cpu_manager.go:414] "containerMap: removing container" podUID="119e2bed-7181-48bc-a8f2-8be78a229cee" containerName="payload-pre-processing"
Oct 31 12:48:45.473374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.437723    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119e2bed-7181-48bc-a8f2-8be78a229cee" containerName="payload-pre-processing"
Oct 31 12:48:45.473374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.438594    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="119e2bed-7181-48bc-a8f2-8be78a229cee" containerName="payload-pre-processing"
Oct 31 12:48:45.542248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.542215    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:45.549809 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.549778    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ad1af7-serving-cert\""
Oct 31 12:48:45.699375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.698960    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"]
Oct 31 12:48:45.778428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.738726    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-metrics-certs\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:45.778428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.738804    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r988s\" (UniqueName: \"kubernetes.io/projected/330b17f2-d442-4686-aec9-81654f1d01b2-kube-api-access-r988s\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:45.778428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.738978    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:45.778428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.739106    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/330b17f2-d442-4686-aec9-81654f1d01b2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:45.844789 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.844744    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-metrics-certs\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:45.845861 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.845819    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r988s\" (UniqueName: \"kubernetes.io/projected/330b17f2-d442-4686-aec9-81654f1d01b2-kube-api-access-r988s\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:45.846874 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.846845    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:45.847220 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:45.847086    5663 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-serving-cert: secret "maas-api-e2e-inf-b-ad1af7-serving-cert" not found
Oct 31 12:48:45.852800 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:45.852771    5663 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls podName:330b17f2-d442-4686-aec9-81654f1d01b2 nodeName:}" failed. No retries permitted until 2026-10-31 12:48:46.352741403 +0000 UTC m=+1251.093167597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls") pod "maas-api-e2e-inf-b-ad1af7-875478959-h5kqf" (UID: "330b17f2-d442-4686-aec9-81654f1d01b2") : secret "maas-api-e2e-inf-b-ad1af7-serving-cert" not found
Oct 31 12:48:45.853058 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.853037    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/330b17f2-d442-4686-aec9-81654f1d01b2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:45.867827 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.865982    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-metrics-certs\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:45.868267 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.867670    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/330b17f2-d442-4686-aec9-81654f1d01b2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:45.890420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:45.890369    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r988s\" (UniqueName: \"kubernetes.io/projected/330b17f2-d442-4686-aec9-81654f1d01b2-kube-api-access-r988s\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:46.390565 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:46.390333    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:46.391120 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:46.390620    5663 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-serving-cert: secret "maas-api-e2e-inf-b-ad1af7-serving-cert" not found
Oct 31 12:48:46.391120 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:46.390710    5663 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls podName:330b17f2-d442-4686-aec9-81654f1d01b2 nodeName:}" failed. No retries permitted until 2026-10-31 12:48:47.390690958 +0000 UTC m=+1252.131117166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls") pod "maas-api-e2e-inf-b-ad1af7-875478959-h5kqf" (UID: "330b17f2-d442-4686-aec9-81654f1d01b2") : secret "maas-api-e2e-inf-b-ad1af7-serving-cert" not found
Oct 31 12:48:46.809202 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:46.807670    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"]
Oct 31 12:48:46.814811 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:46.814774    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"]
Oct 31 12:48:47.466084 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:47.466040    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:47.762085 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:47.759633    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls\") pod \"maas-api-e2e-inf-b-ad1af7-875478959-h5kqf\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:48.035137 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:48.035039    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:48.391107 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:48.391066    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"]
Oct 31 12:48:48.393790 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:48:48.393761    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330b17f2_d442_4686_aec9_81654f1d01b2.slice/crio-bc338bf6ae2d951594172c37d6322ad152fdb58f902cddc755495cdd7047752e WatchSource:0}: Error finding container bc338bf6ae2d951594172c37d6322ad152fdb58f902cddc755495cdd7047752e: Status 404 returned error can't find the container with id bc338bf6ae2d951594172c37d6322ad152fdb58f902cddc755495cdd7047752e
Oct 31 12:48:48.395948 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:48.395922    5663 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:48:48.678005 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:48.677924    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf" event={"ID":"330b17f2-d442-4686-aec9-81654f1d01b2","Type":"ContainerStarted","Data":"bc338bf6ae2d951594172c37d6322ad152fdb58f902cddc755495cdd7047752e"}
Oct 31 12:48:48.767562 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:48.767513    5663 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 12:48:48.767774 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:48.767578    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:48:49.788374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:49.780850    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf" event={"ID":"330b17f2-d442-4686-aec9-81654f1d01b2","Type":"ContainerStarted","Data":"b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3"}
Oct 31 12:48:49.788374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:49.781512    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:49.979893 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:49.979827    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf" podStartSLOduration=5.523871291 podStartE2EDuration="5.979811677s" totalImagesPullingTime="455.940386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:48:49.853807273 +0000 UTC m=+1254.594233513" watchObservedRunningTime="2026-10-31 12:48:49.979811677 +0000 UTC m=+1254.720237893"
Oct 31 12:48:51.808272 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:51.808229    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh" podUID="d46f32c2-2bd4-46b3-bb3c-0a075e1f483a" containerName="payload-processing" containerID="cri-o://8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e" gracePeriod=25
Oct 31 12:48:51.816038 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:51.816008    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p" podUID="1e307b76-8afc-4b60-9996-a980104474ef" containerName="payload-pre-processing" containerID="cri-o://8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc" gracePeriod=25
Oct 31 12:48:52.139462 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.139415    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"
Oct 31 12:48:52.335216 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.302282    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-plugins-config-volume\") pod \"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a\" (UID: \"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a\") "
Oct 31 12:48:52.335764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.335676    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-kube-api-access-pzllv\" (UniqueName: \"kubernetes.io/projected/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-kube-api-access-pzllv\") pod \"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a\" (UID: \"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a\") "
Oct 31 12:48:52.337808 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.337776    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-plugins-config-volume" pod "d46f32c2-2bd4-46b3-bb3c-0a075e1f483a" (UID: "d46f32c2-2bd4-46b3-bb3c-0a075e1f483a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:48:52.345015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.344993    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"
Oct 31 12:48:52.353003 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.352970    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-kube-api-access-pzllv" pod "d46f32c2-2bd4-46b3-bb3c-0a075e1f483a" (UID: "d46f32c2-2bd4-46b3-bb3c-0a075e1f483a"). InnerVolumeSpecName "kube-api-access-pzllv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:52.438731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.438700    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e307b76-8afc-4b60-9996-a980104474ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e307b76-8afc-4b60-9996-a980104474ef-plugins-config-volume\") pod \"1e307b76-8afc-4b60-9996-a980104474ef\" (UID: \"1e307b76-8afc-4b60-9996-a980104474ef\") "
Oct 31 12:48:52.438924 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.438829    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e307b76-8afc-4b60-9996-a980104474ef-kube-api-access-drgj4\" (UniqueName: \"kubernetes.io/projected/1e307b76-8afc-4b60-9996-a980104474ef-kube-api-access-drgj4\") pod \"1e307b76-8afc-4b60-9996-a980104474ef\" (UID: \"1e307b76-8afc-4b60-9996-a980104474ef\") "
Oct 31 12:48:52.439064 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.439041    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e307b76-8afc-4b60-9996-a980104474ef-plugins-config-volume" pod "1e307b76-8afc-4b60-9996-a980104474ef" (UID: "1e307b76-8afc-4b60-9996-a980104474ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:48:52.439545 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.439523    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:52.439658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.439546    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e307b76-8afc-4b60-9996-a980104474ef-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:52.439658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.439562    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzllv\" (UniqueName: \"kubernetes.io/projected/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a-kube-api-access-pzllv\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:52.441013 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.440993    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e307b76-8afc-4b60-9996-a980104474ef-kube-api-access-drgj4" pod "1e307b76-8afc-4b60-9996-a980104474ef" (UID: "1e307b76-8afc-4b60-9996-a980104474ef"). InnerVolumeSpecName "kube-api-access-drgj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:48:52.486534 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:52.486421    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:48:52.542327 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.542181    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drgj4\" (UniqueName: \"kubernetes.io/projected/1e307b76-8afc-4b60-9996-a980104474ef-kube-api-access-drgj4\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:48:52.985451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.972571    5663 generic.go:356] "Generic (PLEG): container finished" podID="1e307b76-8afc-4b60-9996-a980104474ef" containerID="8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc" exitCode=0
Oct 31 12:48:52.985451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.972727    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p" event={"ID":"1e307b76-8afc-4b60-9996-a980104474ef","Type":"ContainerDied","Data":"8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc"}
Oct 31 12:48:52.985451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.972783    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p" event={"ID":"1e307b76-8afc-4b60-9996-a980104474ef","Type":"ContainerDied","Data":"0dfd989e31bf303d6a1bf3689144a8476ca08dacf6e1a2c490a82de187116095"}
Oct 31 12:48:52.985451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.972811    5663 scope.go:122] "RemoveContainer" containerID="8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc"
Oct 31 12:48:52.985451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:52.972993    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"
Oct 31 12:48:53.073731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.071686    5663 generic.go:356] "Generic (PLEG): container finished" podID="d46f32c2-2bd4-46b3-bb3c-0a075e1f483a" containerID="8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e" exitCode=0
Oct 31 12:48:53.073731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.071838    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh" event={"ID":"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a","Type":"ContainerDied","Data":"8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e"}
Oct 31 12:48:53.073731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.071893    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh" event={"ID":"d46f32c2-2bd4-46b3-bb3c-0a075e1f483a","Type":"ContainerDied","Data":"d76a11f0b1797291758f3fcc3b97434874514f58e869728266830da84f3327f2"}
Oct 31 12:48:53.073731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.071985    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"
Oct 31 12:48:53.135683 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.130743    5663 scope.go:122] "RemoveContainer" containerID="8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc"
Oct 31 12:48:53.135683 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:53.132692    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc\": container with ID starting with 8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc not found: ID does not exist" containerID="8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc"
Oct 31 12:48:53.135683 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.132727    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc"} err="failed to get container status \"8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc\": rpc error: code = NotFound desc = could not find container \"8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc\": container with ID starting with 8fdd84e48bcee243f6f934e50c95d0ebe27e6f68e64a6670bc1e7bf104f5fecc not found: ID does not exist"
Oct 31 12:48:53.135683 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.132750    5663 scope.go:122] "RemoveContainer" containerID="8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e"
Oct 31 12:48:53.135683 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.133932    5663 generic.go:356] "Generic (PLEG): container finished" podID="22e10166-45ce-4baa-97b9-6b1544701e43" containerID="06624e83e2d7d3ec10dda92fc212e3731fe48d9f34fd1cb2b767bf0b148c6150" exitCode=0
Oct 31 12:48:53.135683 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.133986    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" event={"ID":"22e10166-45ce-4baa-97b9-6b1544701e43","Type":"ContainerDied","Data":"06624e83e2d7d3ec10dda92fc212e3731fe48d9f34fd1cb2b767bf0b148c6150"}
Oct 31 12:48:53.135683 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.134530    5663 scope.go:122] "RemoveContainer" containerID="06624e83e2d7d3ec10dda92fc212e3731fe48d9f34fd1cb2b767bf0b148c6150"
Oct 31 12:48:53.146629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.142133    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:53.146629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.142238    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:53.146629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.142292    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:48:53.160381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.159491    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"]
Oct 31 12:48:53.185377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.184284    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"
Oct 31 12:48:53.192482 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.192441    5663 scope.go:122] "RemoveContainer" containerID="8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e"
Oct 31 12:48:53.195263 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:48:53.195108    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e\": container with ID starting with 8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e not found: ID does not exist" containerID="8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e"
Oct 31 12:48:53.195263 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.195147    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e"} err="failed to get container status \"8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e\": rpc error: code = NotFound desc = could not find container \"8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e\": container with ID starting with 8c611b7d64a5100775217614a716ee795f086717fefadba73d5faeba821d4d1e not found: ID does not exist"
Oct 31 12:48:53.196550 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.195586    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"
Oct 31 12:48:53.207193 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.207123    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e1fc0ffe-6cf5db6567-fbq9p"]
Oct 31 12:48:53.287624 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.287596    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"]
Oct 31 12:48:53.296550 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.296424    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1fc0ffe-58b9d4b5bc-cbrlh"]
Oct 31 12:48:53.532156 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.532051    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e307b76-8afc-4b60-9996-a980104474ef" path="/var/lib/kubelet/pods/1e307b76-8afc-4b60-9996-a980104474ef/volumes"
Oct 31 12:48:53.533105 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:53.533059    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46f32c2-2bd4-46b3-bb3c-0a075e1f483a" path="/var/lib/kubelet/pods/d46f32c2-2bd4-46b3-bb3c-0a075e1f483a/volumes"
Oct 31 12:48:54.175377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:54.163619    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" event={"ID":"22e10166-45ce-4baa-97b9-6b1544701e43","Type":"ContainerStarted","Data":"ed5f71d3b2bc6deb51b980bdf7c6454fd560f9773e5b9d1447c5ef06f6e995d0"}
Oct 31 12:48:55.078380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:55.074767    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:48:55.078380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:55.078145    5663 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 12:48:55.078380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:55.078203    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 12:48:55.795705 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:55.795680    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:48:56.082215 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:56.082169    5663 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 12:48:56.082478 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:56.082243    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 12:48:56.246391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:56.246341    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"]
Oct 31 12:48:57.079831 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:57.079794    5663 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 12:48:57.080553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:57.080514    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 12:48:58.095646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:58.095605    5663 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 12:48:58.096133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:58.095673    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 12:48:59.081979 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:59.081922    5663 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 12:48:59.082199 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:48:59.081993    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 12:49:00.070290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:00.069791    5663 patch_prober.go:29] interesting pod/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 12:49:00.070290 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:00.069877    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 12:49:01.071769 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:01.071735    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:49:01.072256 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:01.072237    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:49:01.073050 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:01.073027    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:49:06.925553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:06.916122    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"]
Oct 31 12:49:06.925553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:06.916479    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy" containerID="cri-o://56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838" gracePeriod=30
Oct 31 12:49:08.736305 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.736265    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"]
Oct 31 12:49:08.737236 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.737215    5663 cpu_manager.go:414] "containerMap: removing container" podUID="1e307b76-8afc-4b60-9996-a980104474ef" containerName="payload-pre-processing"
Oct 31 12:49:08.737344 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.737237    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e307b76-8afc-4b60-9996-a980104474ef" containerName="payload-pre-processing"
Oct 31 12:49:08.737344 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.737264    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d46f32c2-2bd4-46b3-bb3c-0a075e1f483a" containerName="payload-processing"
Oct 31 12:49:08.737344 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.737271    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46f32c2-2bd4-46b3-bb3c-0a075e1f483a" containerName="payload-processing"
Oct 31 12:49:08.737568 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.737529    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e307b76-8afc-4b60-9996-a980104474ef" containerName="payload-pre-processing"
Oct 31 12:49:08.737568 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.737546    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46f32c2-2bd4-46b3-bb3c-0a075e1f483a" containerName="payload-processing"
Oct 31 12:49:08.741910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.741887    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.749733 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.749705    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-015701\"/\"kube-root-ca.crt\""
Oct 31 12:49:08.749898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.749831    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-015701\"/\"test-model-015701-kserve-self-signed-certs\""
Oct 31 12:49:08.751143 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.751117    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-015701\"/\"openshift-service-ca.crt\""
Oct 31 12:49:08.763466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.763418    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"]
Oct 31 12:49:08.777822 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.777789    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-015701\"/\"default-dockercfg-cq6rh\""
Oct 31 12:49:08.878688 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.878652    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc6sk\" (UniqueName: \"kubernetes.io/projected/2b7f4674-4856-4c0a-820c-24a59546c885-kube-api-access-zc6sk\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.878871 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.878820    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7f4674-4856-4c0a-820c-24a59546c885-tls-certs\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.878986 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.878960    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-home\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.879025 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.879009    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-dshm\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.879256 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.879238    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-tmp-dir\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.879309 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.879296    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-model-cache\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.994338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.989788    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-tmp-dir\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.994338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.989867    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-model-cache\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.994338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.990020    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc6sk\" (UniqueName: \"kubernetes.io/projected/2b7f4674-4856-4c0a-820c-24a59546c885-kube-api-access-zc6sk\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.994338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.990119    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7f4674-4856-4c0a-820c-24a59546c885-tls-certs\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.994338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.990198    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-home\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.994338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.990227    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-dshm\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.994338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.992036    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-tmp-dir\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:08.994338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:08.992310    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-model-cache\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:09.009817 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:09.001053    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-home\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:09.017682 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:09.010694    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7f4674-4856-4c0a-820c-24a59546c885-tls-certs\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:09.047263 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:09.043177    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc6sk\" (UniqueName: \"kubernetes.io/projected/2b7f4674-4856-4c0a-820c-24a59546c885-kube-api-access-zc6sk\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:09.072379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:09.066847    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-dshm\") pod \"test-model-015701-kserve-6b595fcd55-b4ff4\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") " pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:09.103381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:09.101820    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:09.710141 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:09.710055    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"]
Oct 31 12:49:09.723655 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:49:09.723625    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b7f4674_4856_4c0a_820c_24a59546c885.slice/crio-0c86cce6445ce46000c9b948459d395f0ff34e8eb5fd207392433ff8d39e5ef3 WatchSource:0}: Error finding container 0c86cce6445ce46000c9b948459d395f0ff34e8eb5fd207392433ff8d39e5ef3: Status 404 returned error can't find the container with id 0c86cce6445ce46000c9b948459d395f0ff34e8eb5fd207392433ff8d39e5ef3
Oct 31 12:49:10.702645 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:10.702606    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4" event={"ID":"2b7f4674-4856-4c0a-820c-24a59546c885","Type":"ContainerStarted","Data":"44aad1c4b157417bbad9a984ccca570d7863354614a3a85163c4c984f2e8a8b6"}
Oct 31 12:49:10.703061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:10.702656    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4" event={"ID":"2b7f4674-4856-4c0a-820c-24a59546c885","Type":"ContainerStarted","Data":"0c86cce6445ce46000c9b948459d395f0ff34e8eb5fd207392433ff8d39e5ef3"}
Oct 31 12:49:11.114381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.111160    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:11.322375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.322229    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w" podUID="ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" containerName="main" containerID="cri-o://0369126dc99163ed08427e6f7c0189c1e2011c3362afdcf61d66353773e30196" gracePeriod=45
Oct 31 12:49:11.661803 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.661765    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w" podUID="ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": dial tcp 10.133.0.99:8000: connect: connection refused"
Oct 31 12:49:11.750166 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.749040    5663 generic.go:356] "Generic (PLEG): container finished" podID="ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" containerID="0369126dc99163ed08427e6f7c0189c1e2011c3362afdcf61d66353773e30196" exitCode=0
Oct 31 12:49:11.750166 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.750084    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w" event={"ID":"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e","Type":"ContainerDied","Data":"0369126dc99163ed08427e6f7c0189c1e2011c3362afdcf61d66353773e30196"}
Oct 31 12:49:11.858972 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.858942    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:49:11.880397 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.880312    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4" podStartSLOduration=3.880297382 podStartE2EDuration="3.880297382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:10.726638819 +0000 UTC m=+1275.467065036" watchObservedRunningTime="2026-10-31 12:49:11.880297382 +0000 UTC m=+1276.620723598"
Oct 31 12:49:11.948136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.948042    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-model-cache\") pod \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") "
Oct 31 12:49:11.948136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.948100    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-home\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-home\") pod \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") "
Oct 31 12:49:11.948403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.948163    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-kube-api-access-q6g9f\" (UniqueName: \"kubernetes.io/projected/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-kube-api-access-q6g9f\") pod \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") "
Oct 31 12:49:11.948403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.948233    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tls-certs\") pod \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") "
Oct 31 12:49:11.948531 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.948474    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tmp-dir\") pod \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") "
Oct 31 12:49:11.948588 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.948546    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-dshm\") pod \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\" (UID: \"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e\") "
Oct 31 12:49:11.948700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.948460    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-model-cache" pod "ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" (UID: "ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:11.948795 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.948522    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-home" pod "ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" (UID: "ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:11.949374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.949331    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tmp-dir" pod "ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" (UID: "ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:11.949480 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.949440    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:11.949480 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.949459    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:11.951724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.951695    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-kube-api-access-q6g9f" pod "ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" (UID: "ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e"). InnerVolumeSpecName "kube-api-access-q6g9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:49:11.954027 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.953997    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-dshm" pod "ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" (UID: "ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:11.954799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:11.954770    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tls-certs" pod "ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" (UID: "ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:49:12.051637 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.051605    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6g9f\" (UniqueName: \"kubernetes.io/projected/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-kube-api-access-q6g9f\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.051637 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.051636    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.051835 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.051648    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.051835 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.051658    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.684197 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.684175    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:49:12.753026 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.752940    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"]
Oct 31 12:49:12.754105 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.754077    5663 cpu_manager.go:414] "containerMap: removing container" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy"
Oct 31 12:49:12.754105 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.754105    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy"
Oct 31 12:49:12.754289 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.754147    5663 cpu_manager.go:414] "containerMap: removing container" podUID="ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" containerName="main"
Oct 31 12:49:12.754289 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.754156    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" containerName="main"
Oct 31 12:49:12.754289 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.754186    5663 cpu_manager.go:414] "containerMap: removing container" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy"
Oct 31 12:49:12.754289 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.754195    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy"
Oct 31 12:49:12.754521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.754388    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" containerName="main"
Oct 31 12:49:12.754521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.754411    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy"
Oct 31 12:49:12.758148 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.758125    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:12.765861 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.765811    5663 generic.go:356] "Generic (PLEG): container finished" podID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerID="56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838" exitCode=0
Oct 31 12:49:12.766104 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.765857    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" event={"ID":"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0","Type":"ContainerDied","Data":"56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838"}
Oct 31 12:49:12.766235 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.766145    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc" event={"ID":"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0","Type":"ContainerDied","Data":"a33ec74c7d0f72d57d21bb034f60409cc70fe42f52e49b90e708e942fae2ea17"}
Oct 31 12:49:12.766235 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.766182    5663 scope.go:122] "RemoveContainer" containerID="56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838"
Oct 31 12:49:12.766478 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.765892    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"
Oct 31 12:49:12.773825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.773763    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4e8eb9-serving-cert\""
Oct 31 12:49:12.775432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.775040    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-credential-socket\") pod \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") "
Oct 31 12:49:12.775432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.775074    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-socket\") pod \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") "
Oct 31 12:49:12.775432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.775163    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-certs\") pod \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") "
Oct 31 12:49:12.775432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.775328    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-kube-api-access-nrqdj\" (UniqueName: \"kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-kube-api-access-nrqdj\") pod \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") "
Oct 31 12:49:12.775432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.775383    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-envoy\") pod \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") "
Oct 31 12:49:12.775905 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.775452    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-podinfo\") pod \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") "
Oct 31 12:49:12.775905 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.775488    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istiod-ca-cert\") pod \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") "
Oct 31 12:49:12.775905 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.775542    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-data\") pod \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") "
Oct 31 12:49:12.775905 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.775664    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-token\" (UniqueName: \"kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-token\") pod \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\" (UID: \"7e24dd92-1f9a-4bd5-9451-9bc8d05064d0\") "
Oct 31 12:49:12.785504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.780660    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-socket" pod "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" (UID: "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:12.785504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.780928    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-credential-socket" pod "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" (UID: "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:12.785504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.781121    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-certs" pod "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" (UID: "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:12.785504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.785084    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"]
Oct 31 12:49:12.785504 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.785479    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-data" pod "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" (UID: "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:12.785828 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.785603    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-token" pod "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" (UID: "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:49:12.785828 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.785719    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-kube-api-access-nrqdj" pod "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" (UID: "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0"). InnerVolumeSpecName "kube-api-access-nrqdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:49:12.785937 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.785849    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istiod-ca-cert" pod "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" (UID: "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:49:12.786249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.786140    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"
Oct 31 12:49:12.786452 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.786392    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w" event={"ID":"ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e","Type":"ContainerDied","Data":"1f7eeaea1f6de90546c80efc6eed01ce0656bbf19f2530756acd3c5ab946ee0d"}
Oct 31 12:49:12.810457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.804794    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-envoy" pod "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" (UID: "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:12.814231 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.814196    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-podinfo" pod "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" (UID: "7e24dd92-1f9a-4bd5-9451-9bc8d05064d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:49:12.832151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.832128    5663 scope.go:122] "RemoveContainer" containerID="118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d"
Oct 31 12:49:12.845254 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.845230    5663 scope.go:122] "RemoveContainer" containerID="56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838"
Oct 31 12:49:12.845577 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:49:12.845555    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838\": container with ID starting with 56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838 not found: ID does not exist" containerID="56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838"
Oct 31 12:49:12.845664 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.845590    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838"} err="failed to get container status \"56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838\": rpc error: code = NotFound desc = could not find container \"56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838\": container with ID starting with 56bd21ec75e4be990948438b649258bf5c4649cd968730c430199709733a5838 not found: ID does not exist"
Oct 31 12:49:12.845664 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.845627    5663 scope.go:122] "RemoveContainer" containerID="118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d"
Oct 31 12:49:12.845962 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:49:12.845939    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d\": container with ID starting with 118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d not found: ID does not exist" containerID="118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d"
Oct 31 12:49:12.846001 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.845969    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d"} err="failed to get container status \"118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d\": rpc error: code = NotFound desc = could not find container \"118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d\": container with ID starting with 118f455eef13c455398d901809499d13373914c7b4d295ae192f35bb7b8a9f3d not found: ID does not exist"
Oct 31 12:49:12.846001 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.845985    5663 scope.go:122] "RemoveContainer" containerID="0369126dc99163ed08427e6f7c0189c1e2011c3362afdcf61d66353773e30196"
Oct 31 12:49:12.846604 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.846586    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"]
Oct 31 12:49:12.853206 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.853183    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e1fc0ffe/e2e-lifecycle-model-e1fc0ffe-kserve-6c48785497-4k57w"]
Oct 31 12:49:12.877918 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.877887    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-maas-api-tls\") pod \"maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:12.878024 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.877941    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nl48\" (UniqueName: \"kubernetes.io/projected/8729e399-98e5-4087-bb60-2479bd32e646-kube-api-access-5nl48\") pod \"maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:12.878024 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.877968    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-metrics-certs\") pod \"maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:12.878162 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.878121    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8729e399-98e5-4087-bb60-2479bd32e646-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:12.878414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.878395    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrqdj\" (UniqueName: \"kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-kube-api-access-nrqdj\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.878502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.878418    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.878502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.878433    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.878502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.878447    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.878502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.878460    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.878502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.878474    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.878502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.878488    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.878502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.878501    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.878727 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.878514    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:12.979960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.979925    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-maas-api-tls\") pod \"maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:12.980146 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.979974    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nl48\" (UniqueName: \"kubernetes.io/projected/8729e399-98e5-4087-bb60-2479bd32e646-kube-api-access-5nl48\") pod \"maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:12.980146 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.980002    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-metrics-certs\") pod \"maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:12.980146 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.980061    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8729e399-98e5-4087-bb60-2479bd32e646-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:12.980770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.980740    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8729e399-98e5-4087-bb60-2479bd32e646-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:12.982560 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.982536    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-metrics-certs\") pod \"maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:12.982653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.982569    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-maas-api-tls\") pod \"maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:12.992093 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:12.992062    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nl48\" (UniqueName: \"kubernetes.io/projected/8729e399-98e5-4087-bb60-2479bd32e646-kube-api-access-5nl48\") pod \"maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:13.090675 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:13.090650    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:13.117463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:13.117435    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"]
Oct 31 12:49:13.168407 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:13.162655    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5c5815-openshift-default-d5cb6555-vrdzc"]
Oct 31 12:49:13.387783 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:13.387745    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"]
Oct 31 12:49:13.389177 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:49:13.389151    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8729e399_98e5_4087_bb60_2479bd32e646.slice/crio-eeb85889e6ad72458bab8c4862ace00cf179c39cf61020ccb0d287685c3ca5d6 WatchSource:0}: Error finding container eeb85889e6ad72458bab8c4862ace00cf179c39cf61020ccb0d287685c3ca5d6: Status 404 returned error can't find the container with id eeb85889e6ad72458bab8c4862ace00cf179c39cf61020ccb0d287685c3ca5d6
Oct 31 12:49:13.442292 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:13.442260    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" path="/var/lib/kubelet/pods/7e24dd92-1f9a-4bd5-9451-9bc8d05064d0/volumes"
Oct 31 12:49:13.442910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:13.442885    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e" path="/var/lib/kubelet/pods/ca648d2f-1d1e-4ccb-a541-e1d6e7a5409e/volumes"
Oct 31 12:49:13.795246 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:13.795165    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz" event={"ID":"8729e399-98e5-4087-bb60-2479bd32e646","Type":"ContainerStarted","Data":"eeb85889e6ad72458bab8c4862ace00cf179c39cf61020ccb0d287685c3ca5d6"}
Oct 31 12:49:14.805892 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:14.805855    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz" event={"ID":"8729e399-98e5-4087-bb60-2479bd32e646","Type":"ContainerStarted","Data":"5c05a69412f2d20505069be6bbb6fb1f9a78d68f18219fe4f80fa3fa85ce755f"}
Oct 31 12:49:14.806281 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:14.805962    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:14.832835 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:14.832778    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz" podStartSLOduration=15.416146108 podStartE2EDuration="15.832760307s" totalImagesPullingTime="416.614199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:14.829497716 +0000 UTC m=+1279.569923948" watchObservedRunningTime="2026-10-31 12:49:14.832760307 +0000 UTC m=+1279.573186529"
Oct 31 12:49:17.914851 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:17.914813    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"]
Oct 31 12:49:19.104382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:19.103843    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:19.171224 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:19.170851    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:20.011786 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:20.011756    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:20.847661 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:20.847629    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:49:22.480741 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:49:22.480695    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:49:22.927540 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:22.927494    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg" podUID="d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9" containerName="payload-processing" containerID="cri-o://5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d" gracePeriod=25
Oct 31 12:49:22.943541 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:49:22.943479    5663 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40fca5d_2ca4_4ea7_9e33_59dfe670cbb9.slice/crio-conmon-5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d.scope/memory.swap.max": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40fca5d_2ca4_4ea7_9e33_59dfe670cbb9.slice/crio-conmon-5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d.scope/memory.swap.max: no such device
Oct 31 12:49:23.058125 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:49:23.058093    5663 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40fca5d_2ca4_4ea7_9e33_59dfe670cbb9.slice/crio-conmon-5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:49:23.244863 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.244841    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"
Oct 31 12:49:23.275762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.275731    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"]
Oct 31 12:49:23.276561 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.276536    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9" containerName="payload-processing"
Oct 31 12:49:23.276561 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.276560    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9" containerName="payload-processing"
Oct 31 12:49:23.276770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.276687    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9" containerName="payload-processing"
Oct 31 12:49:23.276770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.276701    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e24dd92-1f9a-4bd5-9451-9bc8d05064d0" containerName="istio-proxy"
Oct 31 12:49:23.280856 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.280831    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.281625 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.281592    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-plugins-config-volume\") pod \"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9\" (UID: \"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9\") "
Oct 31 12:49:23.281858 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.281764    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-kube-api-access-cn9vj\" (UniqueName: \"kubernetes.io/projected/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-kube-api-access-cn9vj\") pod \"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9\" (UID: \"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9\") "
Oct 31 12:49:23.282258 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.282123    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-plugins-config-volume" pod "d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9" (UID: "d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:49:23.282628 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.282604    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:23.284713 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.284680    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-kube-api-access-cn9vj" pod "d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9" (UID: "d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9"). InnerVolumeSpecName "kube-api-access-cn9vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:49:23.327108 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.327077    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"]
Oct 31 12:49:23.384520 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.384486    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-token\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.384709 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.384688    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v4pg\" (UniqueName: \"kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-kube-api-access-2v4pg\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.384748 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.384722    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-podinfo\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.384815 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.384805    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istiod-ca-cert\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.384849 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.384832    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-certs\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.384882 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.384872    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-credential-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.385055 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.385034    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-envoy\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.385093 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.385075    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-data\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.385306 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.385123    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.385306 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.385248    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn9vj\" (UniqueName: \"kubernetes.io/projected/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9-kube-api-access-cn9vj\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:23.486130 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486064    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-certs\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.486130 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486121    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-credential-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.486614 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486202    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-envoy\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.486614 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486241    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-data\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.486614 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486275    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.486614 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486376    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-token\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.486614 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486526    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-certs\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.486614 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486554    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-credential-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.486896 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486636    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v4pg\" (UniqueName: \"kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-kube-api-access-2v4pg\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.486896 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486671    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-podinfo\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.486896 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486701    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-socket\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.486896 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486767    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istiod-ca-cert\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.486896 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.486841    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-data\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.487437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.487413    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istiod-ca-cert\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.488629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.488609    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-envoy\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.489003 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.488986    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-podinfo\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.498932 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.498905    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v4pg\" (UniqueName: \"kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-kube-api-access-2v4pg\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.499215 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.499198    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-token\") pod \"e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") " pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:23.623638 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:23.623599    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:24.045875 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.045807    5663 generic.go:356] "Generic (PLEG): container finished" podID="d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9" containerID="5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d" exitCode=0
Oct 31 12:49:24.046034 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.045951    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg" event={"ID":"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9","Type":"ContainerDied","Data":"5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d"}
Oct 31 12:49:24.046034 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.045996    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg" event={"ID":"d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9","Type":"ContainerDied","Data":"59aa9b87fbf4f5c47b51bc552eb8027d9954f510362db958f126b73a9b43917f"}
Oct 31 12:49:24.046034 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.046017    5663 scope.go:122] "RemoveContainer" containerID="5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d"
Oct 31 12:49:24.046197 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.046181    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"
Oct 31 12:49:24.057869 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.057849    5663 scope.go:122] "RemoveContainer" containerID="5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d"
Oct 31 12:49:24.058148 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:49:24.058127    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d\": container with ID starting with 5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d not found: ID does not exist" containerID="5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d"
Oct 31 12:49:24.058200 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.058155    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d"} err="failed to get container status \"5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d\": rpc error: code = NotFound desc = could not find container \"5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d\": container with ID starting with 5535cb423dbe9d2f9873d9245cfde5a1bf95dfd11c17218ea7237ec3e319e65d not found: ID does not exist"
Oct 31 12:49:24.072481 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:49:24.072450    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd903ed7b_b81f_455e_8619_dd1fa515c0aa.slice/crio-90e7623b9fef76845d9b9d3ebd06a8c3f33837a2e698e59eb7138a22b9faa9cd WatchSource:0}: Error finding container 90e7623b9fef76845d9b9d3ebd06a8c3f33837a2e698e59eb7138a22b9faa9cd: Status 404 returned error can't find the container with id 90e7623b9fef76845d9b9d3ebd06a8c3f33837a2e698e59eb7138a22b9faa9cd
Oct 31 12:49:24.074286 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.074264    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"]
Oct 31 12:49:24.076033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.076000    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:24.076149 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.076072    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:24.076149 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.076102    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:24.078934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.078916    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"]
Oct 31 12:49:24.082206 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.082143    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-787316-65b96d4df-vv7rg"]
Oct 31 12:49:24.564555 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.564524    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"]
Oct 31 12:49:24.570269 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.570246    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.580154 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.580009    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ad1af7\"/\"test-model-ad1af7-kserve-self-signed-certs\""
Oct 31 12:49:24.580827 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.580459    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ad1af7\"/\"default-dockercfg-vtn7m\""
Oct 31 12:49:24.582878 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.582857    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"]
Oct 31 12:49:24.586656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.586544    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ad1af7\"/\"openshift-service-ca.crt\""
Oct 31 12:49:24.586656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.586629    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ad1af7\"/\"kube-root-ca.crt\""
Oct 31 12:49:24.643998 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.643959    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-model-cache\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.644317 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.644294    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p9sp\" (UniqueName: \"kubernetes.io/projected/1d224a67-a771-46d2-ad68-c8970ca2dffa-kube-api-access-5p9sp\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.644495 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.644478    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-dshm\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.644646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.644629    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-tmp-dir\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.644959 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.644941    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-home\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.649477 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.649453    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d224a67-a771-46d2-ad68-c8970ca2dffa-tls-certs\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.774521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.774400    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-home\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.774954 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.763904    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-home\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.775297 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.775271    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d224a67-a771-46d2-ad68-c8970ca2dffa-tls-certs\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.803673 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.789524    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-model-cache\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.804085 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.804041    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p9sp\" (UniqueName: \"kubernetes.io/projected/1d224a67-a771-46d2-ad68-c8970ca2dffa-kube-api-access-5p9sp\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.804689 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.804667    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-dshm\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.805191 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.805173    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-tmp-dir\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.805814 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.790168    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-model-cache\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.806007 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.805590    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-tmp-dir\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.816553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.815021    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-dshm\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.848133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.848098    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d224a67-a771-46d2-ad68-c8970ca2dffa-tls-certs\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.885139 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.880528    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p9sp\" (UniqueName: \"kubernetes.io/projected/1d224a67-a771-46d2-ad68-c8970ca2dffa-kube-api-access-5p9sp\") pod \"test-model-ad1af7-kserve-b4db66dc9-b99sn\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") " pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:24.906768 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:24.906461    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:25.071424 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:25.066662    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" event={"ID":"d903ed7b-b81f-455e-8619-dd1fa515c0aa","Type":"ContainerStarted","Data":"fccf4dab745a033cc143e861423ddfecd36feca479dd32476879fb688fd11db2"}
Oct 31 12:49:25.071424 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:25.066732    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" event={"ID":"d903ed7b-b81f-455e-8619-dd1fa515c0aa","Type":"ContainerStarted","Data":"90e7623b9fef76845d9b9d3ebd06a8c3f33837a2e698e59eb7138a22b9faa9cd"}
Oct 31 12:49:25.109803 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:25.109724    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" podStartSLOduration=8.109702526 podStartE2EDuration="8.109702526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:25.103862757 +0000 UTC m=+1289.844288973" watchObservedRunningTime="2026-10-31 12:49:25.109702526 +0000 UTC m=+1289.850128743"
Oct 31 12:49:25.442180 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:25.442104    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9" path="/var/lib/kubelet/pods/d40fca5d-2ca4-4ea7-9e33-59dfe670cbb9/volumes"
Oct 31 12:49:25.491412 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:25.491380    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"]
Oct 31 12:49:25.678388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:25.672510    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:25.678388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:25.675198    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 12:49:25.678388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:25.675257    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" podUID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 12:49:26.087438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:26.086962    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn" event={"ID":"1d224a67-a771-46d2-ad68-c8970ca2dffa","Type":"ContainerStarted","Data":"00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab"}
Oct 31 12:49:26.087438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:26.087014    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn" event={"ID":"1d224a67-a771-46d2-ad68-c8970ca2dffa","Type":"ContainerStarted","Data":"9c033b6ae3eb7ac2a61fd5ad6e15f0e008c546faa0ed9b00fb0255ed61ef71ef"}
Oct 31 12:49:26.107589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:26.107535    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn" podStartSLOduration=2.107521926 podStartE2EDuration="2.107521926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:26.105095193 +0000 UTC m=+1290.845521408" watchObservedRunningTime="2026-10-31 12:49:26.107521926 +0000 UTC m=+1290.847948142"
Oct 31 12:49:26.624263 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:26.624228    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 12:49:26.624428 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:26.624287    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" podUID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 12:49:26.907777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:26.907684    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:27.624100 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:27.624055    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 12:49:27.624306 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:27.624125    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" podUID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 12:49:28.627272 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:28.626082    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 12:49:28.627272 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:28.626137    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" podUID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 12:49:29.624323 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:29.624253    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 12:49:29.624527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:29.624386    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" podUID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 12:49:30.634562 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:30.631204    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 12:49:30.634562 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:30.631263    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" podUID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 12:49:31.629698 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:31.629667    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:31.630030 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:31.630004    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:31.630625 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:31.630598    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:34.491710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:34.491662    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 12:49:34.549387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:34.546865    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:34.619390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:34.586040    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:49:34.684506 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:34.667581    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:49:34.718442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:34.699845    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:34.718442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:34.715670    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:34.762439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:34.761578    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:34.911647 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:34.911480    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:35.097379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.090670    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:35.161424 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.160487    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:35.170385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.169792    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:35.274071 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.274024    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:35.492812 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.492714    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:35.495468 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.494019    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:49:35.580396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.549316    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 31 12:49:35.600303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.581483    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:35.600303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.596334    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:49:35.600303 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.582057    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:35.670775 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.670729    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:35.733405 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.717259    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:49:35.734386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.717392    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:49:35.735246 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.735222    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:49:35.735458 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.735442    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:49:35.740463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.740420    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:49:35.770396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.760202    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:49:35.793503 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.770786    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 31 12:49:35.793503 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.771375    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:49:35.793503 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.788441    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:35.947464 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.917001    5663 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn" podUID="1d224a67-a771-46d2-ad68-c8970ca2dffa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/health\": context deadline exceeded"
Oct 31 12:49:35.963427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:35.957084    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:36.024821 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.024730    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4" podUID="2b7f4674-4856-4c0a-820c-24a59546c885" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded"
Oct 31 12:49:36.122900 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.122870    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:49:36.183380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.173484    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:36.290463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.287451    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:36.290463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.289208    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:49:36.496526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.496477    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 12:49:36.627386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.598515    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:36.627386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.608207    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:36.627386 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.609008    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 31 12:49:36.628888 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.628860    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:49:36.629150 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.629134    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:49:36.711092 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.711042    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:49:36.713542 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.713512    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:49:36.728753 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.728642    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:36.742508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.742453    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 12:49:36.765273 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.765227    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:36.779535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.779488    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:36.810875 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.810829    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:49:36.811820 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.811792    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:49:36.912226 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.912137    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:49:36.932101 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.928534    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 12:49:36.967937 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:36.966997    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:49:37.020374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:37.020121    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:49:37.031495 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:37.031023    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:37.032758 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:37.032725    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4" podUID="2b7f4674-4856-4c0a-820c-24a59546c885" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:37.041811 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:37.033499    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:37.076416 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:37.072364    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:49:37.076724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:37.076704    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:49:37.154548 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:37.154388    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:37.162596 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:37.155017    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:49:37.162596 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:37.156201    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:49:37.162596 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:37.156248    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:49:37.215292 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:37.215258    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:49:44.691429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:44.690338    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"]
Oct 31 12:49:44.691429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:44.690690    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy" containerID="cri-o://aaec2350b02466a7592b86aefe54a97ea0f2c3323f31ebc34c260fefd92e12ba" gracePeriod=30
Oct 31 12:49:44.713517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:44.711900    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"]
Oct 31 12:49:44.713517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:44.712154    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" podUID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" containerName="istio-proxy" containerID="cri-o://fccf4dab745a033cc143e861423ddfecd36feca479dd32476879fb688fd11db2" gracePeriod=30
Oct 31 12:49:45.173455 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:45.171554    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:45.282379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:45.281014    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:46.654375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:46.651549    5663 patch_prober.go:29] interesting pod/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 31 12:49:46.654375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:46.651676    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" podUID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:49:48.025056 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:48.025025    5663 generic.go:356] "Generic (PLEG): container finished" podID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerID="0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58" exitCode=0
Oct 31 12:49:48.025620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:48.025102    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" event={"ID":"9a7ea3d2-2730-4332-8b30-dd2400e2239d","Type":"ContainerDied","Data":"0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58"}
Oct 31 12:49:48.025743 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:48.025724    5663 scope.go:122] "RemoveContainer" containerID="0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58"
Oct 31 12:49:48.028400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:48.027789    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:48.028400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:48.027903    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:48.028400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:48.027948    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:49.035595 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.035559    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" event={"ID":"9a7ea3d2-2730-4332-8b30-dd2400e2239d","Type":"ContainerStarted","Data":"50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250"}
Oct 31 12:49:49.151851 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.151815    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:49:49.153177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.153139    5663 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:49:49.153295 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.153203    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 12:49:49.512886 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.512850    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"]
Oct 31 12:49:49.517663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.517643    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"
Oct 31 12:49:49.521008 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.520987    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:49:49.530535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.530510    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1a7d0e\""
Oct 31 12:49:49.536417 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.536390    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"]
Oct 31 12:49:49.546683 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.546659    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1a7d0e-dockercfg-jtss9\""
Oct 31 12:49:49.588664 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.588616    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5gnp\" (UniqueName: \"kubernetes.io/projected/eba68b32-a972-45c1-aabe-f769ad8113a9-kube-api-access-c5gnp\") pod \"payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q\" (UID: \"eba68b32-a972-45c1-aabe-f769ad8113a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"
Oct 31 12:49:49.588880 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.588769    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba68b32-a972-45c1-aabe-f769ad8113a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q\" (UID: \"eba68b32-a972-45c1-aabe-f769ad8113a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"
Oct 31 12:49:49.693910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.693864    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba68b32-a972-45c1-aabe-f769ad8113a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q\" (UID: \"eba68b32-a972-45c1-aabe-f769ad8113a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"
Oct 31 12:49:49.704100 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.704047    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5gnp\" (UniqueName: \"kubernetes.io/projected/eba68b32-a972-45c1-aabe-f769ad8113a9-kube-api-access-c5gnp\") pod \"payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q\" (UID: \"eba68b32-a972-45c1-aabe-f769ad8113a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"
Oct 31 12:49:49.708400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.708339    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba68b32-a972-45c1-aabe-f769ad8113a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q\" (UID: \"eba68b32-a972-45c1-aabe-f769ad8113a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"
Oct 31 12:49:49.793442 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.783813    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5gnp\" (UniqueName: \"kubernetes.io/projected/eba68b32-a972-45c1-aabe-f769ad8113a9-kube-api-access-c5gnp\") pod \"payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q\" (UID: \"eba68b32-a972-45c1-aabe-f769ad8113a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"
Oct 31 12:49:49.854401 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:49.846124    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"
Oct 31 12:49:50.166578 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.165342    5663 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:49:50.166578 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.165503    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 12:49:50.168057 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.167862    5663 generic.go:356] "Generic (PLEG): container finished" podID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" containerID="fccf4dab745a033cc143e861423ddfecd36feca479dd32476879fb688fd11db2" exitCode=0
Oct 31 12:49:50.168057 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.167997    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" event={"ID":"d903ed7b-b81f-455e-8619-dd1fa515c0aa","Type":"ContainerDied","Data":"fccf4dab745a033cc143e861423ddfecd36feca479dd32476879fb688fd11db2"}
Oct 31 12:49:50.193700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.193659    5663 generic.go:356] "Generic (PLEG): container finished" podID="db35ea34-5527-4982-8a90-3bf05ef57357" containerID="aaec2350b02466a7592b86aefe54a97ea0f2c3323f31ebc34c260fefd92e12ba" exitCode=0
Oct 31 12:49:50.194745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.194716    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" event={"ID":"db35ea34-5527-4982-8a90-3bf05ef57357","Type":"ContainerDied","Data":"aaec2350b02466a7592b86aefe54a97ea0f2c3323f31ebc34c260fefd92e12ba"}
Oct 31 12:49:50.194894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.194781    5663 scope.go:122] "RemoveContainer" containerID="42ae02304d162dfbe913c6a283f7d5c9db63673c03682651c5d70d5b6f5bd2b0"
Oct 31 12:49:50.650770 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:49:50.650739    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba68b32_a972_45c1_aabe_f769ad8113a9.slice/crio-8f66deaf4add3dbf594525614c5d76bc39fc23366ead11941f2e1bc25f2d62c8 WatchSource:0}: Error finding container 8f66deaf4add3dbf594525614c5d76bc39fc23366ead11941f2e1bc25f2d62c8: Status 404 returned error can't find the container with id 8f66deaf4add3dbf594525614c5d76bc39fc23366ead11941f2e1bc25f2d62c8
Oct 31 12:49:50.652546 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.652516    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"]
Oct 31 12:49:50.655825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.655799    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:49:50.745310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.745279    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"]
Oct 31 12:49:50.746628 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.746605    5663 cpu_manager.go:414] "containerMap: removing container" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy"
Oct 31 12:49:50.746628 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.746627    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy"
Oct 31 12:49:50.746923 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.746912    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy"
Oct 31 12:49:50.747005 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.746934    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy"
Oct 31 12:49:50.780646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.755403    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-envoy\") pod \"db35ea34-5527-4982-8a90-3bf05ef57357\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") "
Oct 31 12:49:50.780646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.755488    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-credential-socket\") pod \"db35ea34-5527-4982-8a90-3bf05ef57357\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") "
Oct 31 12:49:50.780646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.755540    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-data\") pod \"db35ea34-5527-4982-8a90-3bf05ef57357\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") "
Oct 31 12:49:50.780646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.755614    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-istio-token\" (UniqueName: \"kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-istio-token\") pod \"db35ea34-5527-4982-8a90-3bf05ef57357\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") "
Oct 31 12:49:50.780646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.755692    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-kube-api-access-5wjvf\" (UniqueName: \"kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-kube-api-access-5wjvf\") pod \"db35ea34-5527-4982-8a90-3bf05ef57357\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") "
Oct 31 12:49:50.780646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.755785    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-certs\") pod \"db35ea34-5527-4982-8a90-3bf05ef57357\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") "
Oct 31 12:49:50.780646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.755882    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db35ea34-5527-4982-8a90-3bf05ef57357-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db35ea34-5527-4982-8a90-3bf05ef57357-istio-podinfo\") pod \"db35ea34-5527-4982-8a90-3bf05ef57357\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") "
Oct 31 12:49:50.780646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.756015    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db35ea34-5527-4982-8a90-3bf05ef57357-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db35ea34-5527-4982-8a90-3bf05ef57357-istiod-ca-cert\") pod \"db35ea34-5527-4982-8a90-3bf05ef57357\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") "
Oct 31 12:49:50.780646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.756043    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-socket\") pod \"db35ea34-5527-4982-8a90-3bf05ef57357\" (UID: \"db35ea34-5527-4982-8a90-3bf05ef57357\") "
Oct 31 12:49:50.780646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.771579    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"]
Oct 31 12:49:50.780646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.776726    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:50.792377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.788941    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4e8eb9-openshift-default-dockercfg-sczrg\""
Oct 31 12:49:50.793327 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.793294    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-certs" pod "db35ea34-5527-4982-8a90-3bf05ef57357" (UID: "db35ea34-5527-4982-8a90-3bf05ef57357"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:50.794226 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.794151    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db35ea34-5527-4982-8a90-3bf05ef57357-istiod-ca-cert" pod "db35ea34-5527-4982-8a90-3bf05ef57357" (UID: "db35ea34-5527-4982-8a90-3bf05ef57357"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:49:50.794456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.794367    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-socket" pod "db35ea34-5527-4982-8a90-3bf05ef57357" (UID: "db35ea34-5527-4982-8a90-3bf05ef57357"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:50.795479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.795316    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.795479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.795342    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db35ea34-5527-4982-8a90-3bf05ef57357-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.795479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.795442    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.797038 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.797013    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-data" pod "db35ea34-5527-4982-8a90-3bf05ef57357" (UID: "db35ea34-5527-4982-8a90-3bf05ef57357"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:50.797775 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.797737    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db35ea34-5527-4982-8a90-3bf05ef57357-istio-podinfo" pod "db35ea34-5527-4982-8a90-3bf05ef57357" (UID: "db35ea34-5527-4982-8a90-3bf05ef57357"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:49:50.798110 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.798079    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-credential-socket" pod "db35ea34-5527-4982-8a90-3bf05ef57357" (UID: "db35ea34-5527-4982-8a90-3bf05ef57357"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:50.803187 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.802489    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-envoy" pod "db35ea34-5527-4982-8a90-3bf05ef57357" (UID: "db35ea34-5527-4982-8a90-3bf05ef57357"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:50.803187 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.802623    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-kube-api-access-5wjvf" pod "db35ea34-5527-4982-8a90-3bf05ef57357" (UID: "db35ea34-5527-4982-8a90-3bf05ef57357"). InnerVolumeSpecName "kube-api-access-5wjvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:49:50.817656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.817629    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-istio-token" pod "db35ea34-5527-4982-8a90-3bf05ef57357" (UID: "db35ea34-5527-4982-8a90-3bf05ef57357"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:49:50.850098 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.850074    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:50.896419 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.896304    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-socket\") pod \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") "
Oct 31 12:49:50.896630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.896411    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-data\") pod \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") "
Oct 31 12:49:50.896630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.896557    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-envoy\") pod \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") "
Oct 31 12:49:50.896630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.896603    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istiod-ca-cert\") pod \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") "
Oct 31 12:49:50.896827 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.896778    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-podinfo\") pod \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") "
Oct 31 12:49:50.897005 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.896976    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-token\" (UniqueName: \"kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-token\") pod \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") "
Oct 31 12:49:50.897096 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.897029    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-credential-socket\") pod \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") "
Oct 31 12:49:50.897096 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.897059    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-certs\") pod \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") "
Oct 31 12:49:50.897239 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.897122    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-kube-api-access-2v4pg\" (UniqueName: \"kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-kube-api-access-2v4pg\") pod \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\" (UID: \"d903ed7b-b81f-455e-8619-dd1fa515c0aa\") "
Oct 31 12:49:50.897579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.897554    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-data" pod "d903ed7b-b81f-455e-8619-dd1fa515c0aa" (UID: "d903ed7b-b81f-455e-8619-dd1fa515c0aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:50.897769 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.897620    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:50.897985 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.897968    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnpg8\" (UniqueName: \"kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-kube-api-access-bnpg8\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:50.899449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.898447    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-certs" pod "d903ed7b-b81f-455e-8619-dd1fa515c0aa" (UID: "d903ed7b-b81f-455e-8619-dd1fa515c0aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:50.899736 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.898821    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-credential-socket" pod "d903ed7b-b81f-455e-8619-dd1fa515c0aa" (UID: "d903ed7b-b81f-455e-8619-dd1fa515c0aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:50.899888 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.899068    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istiod-ca-cert" pod "d903ed7b-b81f-455e-8619-dd1fa515c0aa" (UID: "d903ed7b-b81f-455e-8619-dd1fa515c0aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:49:50.899954 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.899859    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-socket" pod "d903ed7b-b81f-455e-8619-dd1fa515c0aa" (UID: "d903ed7b-b81f-455e-8619-dd1fa515c0aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:50.900013 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.899508    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:50.900151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.900139    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:50.900276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.900266    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:50.900418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.900403    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:50.900894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.900872    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:50.901623 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.901608    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:50.901761 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.901751    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e2d1c0-5688-4db4-9630-f40ce094891e-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:50.901915 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.901907    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.901988 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.901980    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.902043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.902035    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.902105 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.902097    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjvf\" (UniqueName: \"kubernetes.io/projected/db35ea34-5527-4982-8a90-3bf05ef57357-kube-api-access-5wjvf\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.902160 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.902153    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.902211 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.902204    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.902270 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.902262    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db35ea34-5527-4982-8a90-3bf05ef57357-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.902323 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.902316    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.902413 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.902400    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.902529 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.902516    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.902636 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.902624    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db35ea34-5527-4982-8a90-3bf05ef57357-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:50.902880 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.902857    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-envoy" pod "d903ed7b-b81f-455e-8619-dd1fa515c0aa" (UID: "d903ed7b-b81f-455e-8619-dd1fa515c0aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:49:50.904296 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.904269    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-token" pod "d903ed7b-b81f-455e-8619-dd1fa515c0aa" (UID: "d903ed7b-b81f-455e-8619-dd1fa515c0aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:49:50.905345 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.905313    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-podinfo" pod "d903ed7b-b81f-455e-8619-dd1fa515c0aa" (UID: "d903ed7b-b81f-455e-8619-dd1fa515c0aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:49:50.907727 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.907704    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-kube-api-access-2v4pg" pod "d903ed7b-b81f-455e-8619-dd1fa515c0aa" (UID: "d903ed7b-b81f-455e-8619-dd1fa515c0aa"). InnerVolumeSpecName "kube-api-access-2v4pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:49:50.910285 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.910261    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"]
Oct 31 12:49:50.911068 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.911048    5663 cpu_manager.go:414] "containerMap: removing container" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy"
Oct 31 12:49:50.911155 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.911069    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" containerName="istio-proxy"
Oct 31 12:49:50.911155 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.911125    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" containerName="istio-proxy"
Oct 31 12:49:50.911155 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.911135    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" containerName="istio-proxy"
Oct 31 12:49:50.911388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.911340    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" containerName="istio-proxy"
Oct 31 12:49:50.915475 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.915456    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:50.930788 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:50.930762    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"]
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.005502    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.005574    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.005610    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.005648    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.005674    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.005798    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.005837    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.005867    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.005924    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.005960    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e2d1c0-5688-4db4-9630-f40ce094891e-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.005988    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.006221    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.006280    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz4kd\" (UniqueName: \"kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-kube-api-access-kz4kd\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.006326    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnpg8\" (UniqueName: \"kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-kube-api-access-bnpg8\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.006413    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.006474    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.006519    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.006571    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.006758    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.006781    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.006796    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v4pg\" (UniqueName: \"kubernetes.io/projected/d903ed7b-b81f-455e-8619-dd1fa515c0aa-kube-api-access-2v4pg\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:51.007253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.006812    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d903ed7b-b81f-455e-8619-dd1fa515c0aa-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:49:51.008854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.007835    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.008854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.008039    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.008854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.008629    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.008854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.008714    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e2d1c0-5688-4db4-9630-f40ce094891e-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.009562 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.009532    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.012063 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.012037    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.012683 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.012656    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.023148 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.021591    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.023385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.023345    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnpg8\" (UniqueName: \"kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-kube-api-access-bnpg8\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.108202 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.108160    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.108391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.108304    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.108438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.108406    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.108472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.108450    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.108636 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.108617    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-certs\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.108672 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.108629    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz4kd\" (UniqueName: \"kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-kube-api-access-kz4kd\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.108740 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.108718    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.108793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.108749    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.108793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.108782    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.108913 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.108883    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.109065 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.109041    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.109216 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.109193    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-data\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.109278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.109245    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-credential-socket\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.109340 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.109291    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istiod-ca-cert\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.111016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.110991    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-podinfo\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.111106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.111065    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-envoy\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.122109 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.122086    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz4kd\" (UniqueName: \"kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-kube-api-access-kz4kd\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.122234 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.122188    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-token\") pod \"e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") " pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.149226 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.149169    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:51.152348 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.152327    5663 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:49:51.152450 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.152397    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 12:49:51.206240 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.206202    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q" event={"ID":"eba68b32-a972-45c1-aabe-f769ad8113a9","Type":"ContainerStarted","Data":"c6fd56c85e98da1c1b89d98ffc4d26df21702bf17dd3bff3969e398a4085b0a5"}
Oct 31 12:49:51.206663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.206299    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q" event={"ID":"eba68b32-a972-45c1-aabe-f769ad8113a9","Type":"ContainerStarted","Data":"8f66deaf4add3dbf594525614c5d76bc39fc23366ead11941f2e1bc25f2d62c8"}
Oct 31 12:49:51.206663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.206448    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"
Oct 31 12:49:51.208315 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.208296    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"
Oct 31 12:49:51.208470 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.208329    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4" event={"ID":"d903ed7b-b81f-455e-8619-dd1fa515c0aa","Type":"ContainerDied","Data":"90e7623b9fef76845d9b9d3ebd06a8c3f33837a2e698e59eb7138a22b9faa9cd"}
Oct 31 12:49:51.208470 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.208400    5663 scope.go:122] "RemoveContainer" containerID="fccf4dab745a033cc143e861423ddfecd36feca479dd32476879fb688fd11db2"
Oct 31 12:49:51.210725 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.210708    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"
Oct 31 12:49:51.210840 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.210749    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8" event={"ID":"db35ea34-5527-4982-8a90-3bf05ef57357","Type":"ContainerDied","Data":"d2f60ac3271fd66ca0ff307e75cee0e47961a3f8888e7e20df25adaf5cb28c0d"}
Oct 31 12:49:51.229582 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.229552    5663 scope.go:122] "RemoveContainer" containerID="aaec2350b02466a7592b86aefe54a97ea0f2c3323f31ebc34c260fefd92e12ba"
Oct 31 12:49:51.233545 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.233491    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q" podStartSLOduration=2.233475966 podStartE2EDuration="2.233475966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:51.22960751 +0000 UTC m=+1315.970033727" watchObservedRunningTime="2026-10-31 12:49:51.233475966 +0000 UTC m=+1315.973902195"
Oct 31 12:49:51.255063 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.255028    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"]
Oct 31 12:49:51.294949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.294921    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c2wn4"]
Oct 31 12:49:51.299189 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.298973    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:51.307943 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.306073    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"]
Oct 31 12:49:51.314769 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.314744    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e1fc0ffe-openshift-default-5dfdfc44fc-c22f8"]
Oct 31 12:49:51.463066 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.462631    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d903ed7b-b81f-455e-8619-dd1fa515c0aa" path="/var/lib/kubelet/pods/d903ed7b-b81f-455e-8619-dd1fa515c0aa/volumes"
Oct 31 12:49:51.465317 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.465278    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db35ea34-5527-4982-8a90-3bf05ef57357" path="/var/lib/kubelet/pods/db35ea34-5527-4982-8a90-3bf05ef57357/volumes"
Oct 31 12:49:51.516380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.508569    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"]
Oct 31 12:49:51.523091 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:49:51.523060    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e2d1c0_5688_4db4_9630_f40ce094891e.slice/crio-2e8632af93a3d54d6781fd94702d1986ecd1c3c8d8bea7aab72736c7ca52d6bc WatchSource:0}: Error finding container 2e8632af93a3d54d6781fd94702d1986ecd1c3c8d8bea7aab72736c7ca52d6bc: Status 404 returned error can't find the container with id 2e8632af93a3d54d6781fd94702d1986ecd1c3c8d8bea7aab72736c7ca52d6bc
Oct 31 12:49:51.529726 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.529681    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:51.529844 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.529775    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:51.529844 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.529824    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:51.808291 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.802737    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"]
Oct 31 12:49:51.812485 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.812442    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:51.812616 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.812536    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:51.812616 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:51.812579    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:49:52.186073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:52.186036    5663 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:49:52.187443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:52.187409    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 12:49:52.359464 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:52.359425    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" event={"ID":"52e2d1c0-5688-4db4-9630-f40ce094891e","Type":"ContainerStarted","Data":"47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e"}
Oct 31 12:49:52.360076 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:52.360039    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" event={"ID":"52e2d1c0-5688-4db4-9630-f40ce094891e","Type":"ContainerStarted","Data":"2e8632af93a3d54d6781fd94702d1986ecd1c3c8d8bea7aab72736c7ca52d6bc"}
Oct 31 12:49:52.397378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:52.379266    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" event={"ID":"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d","Type":"ContainerStarted","Data":"e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54"}
Oct 31 12:49:52.398407 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:52.398371    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" event={"ID":"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d","Type":"ContainerStarted","Data":"fee6d3cecb00f7224baacc44bd58c0fee7c82bfba1ff329bd9c4a28424403a2b"}
Oct 31 12:49:52.469429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:52.455250    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" podStartSLOduration=5.455229372 podStartE2EDuration="5.455229372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:52.452736147 +0000 UTC m=+1317.193162364" watchObservedRunningTime="2026-10-31 12:49:52.455229372 +0000 UTC m=+1317.195655589"
Oct 31 12:49:52.483283 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:52.483217    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" podStartSLOduration=5.483197857 podStartE2EDuration="5.483197857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:49:52.399492666 +0000 UTC m=+1317.139918887" watchObservedRunningTime="2026-10-31 12:49:52.483197857 +0000 UTC m=+1317.223624075"
Oct 31 12:49:52.707870 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:49:52.707829    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:49:53.178424 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:53.156019    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:49:53.180082 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:53.180033    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 12:49:53.182144 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:53.182119    5663 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:49:53.183952 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:53.183921    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 12:49:53.186381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:53.184718    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:49:53.310378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:53.307632    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:49:53.332507 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:53.332469    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:49:53.332772 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:53.332743    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 12:49:54.219308 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:54.160995    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:49:54.219308 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:54.161058    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 12:49:54.219308 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:54.163076    5663 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:49:54.219308 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:54.163117    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 12:49:54.404454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:54.388630    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:49:54.404454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:54.388802    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 12:49:54.729703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:54.729637    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:54.827380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:54.782064    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:49:55.010886 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:54.984422    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:55.072489 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.072419    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:55.119793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.119752    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:55.139389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.138561    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded"
Oct 31 12:49:55.139659 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.139629    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 31 12:49:55.160369 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.160318    5663 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:49:55.160638 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.160605    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 12:49:55.166565 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.166516    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:55.166980 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.166937    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4" podUID="2b7f4674-4856-4c0a-820c-24a59546c885" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:55.178922 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.174624    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:55.178922 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.175780    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Oct 31 12:49:55.178922 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.175839    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 31 12:49:55.178922 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.176596    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:49:55.178922 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.176637    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 12:49:55.252377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.200652    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn" podUID="1d224a67-a771-46d2-ad68-c8970ca2dffa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:55.253284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.213412    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:55.254302 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.226842    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:49:55.372439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.321172    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:49:55.372439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.321236    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 12:49:55.616296 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.616250    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:55.720526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.720432    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:55.746462 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.746418    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:55.918405 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.918340    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn" podUID="1d224a67-a771-46d2-ad68-c8970ca2dffa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/health\": context deadline exceeded"
Oct 31 12:49:55.955669 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:55.955632    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Oct 31 12:49:56.074387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.073600    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:56.084384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.083496    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:49:56.141385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.122460    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 31 12:49:56.141385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.123071    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:49:56.408447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.223519    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 31 12:49:56.408447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.224454    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:49:56.408447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.226587    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4" podUID="2b7f4674-4856-4c0a-820c-24a59546c885" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:56.408447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.226965    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:56.408447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.233563    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:49:56.408447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.233607    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 12:49:56.408447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.337875    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:56.408447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.362766    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:49:56.430889 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.430852    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:49:56.434373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.431102    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 12:49:56.472089 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.472039    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn" podUID="1d224a67-a771-46d2-ad68-c8970ca2dffa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded"
Oct 31 12:49:56.501756 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.501727    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:56.537460 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.537419    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:56.538068 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.538048    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:49:56.538400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.538378    5663 patch_prober.go:29] interesting pod/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 12:49:56.538566 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.538543    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 12:49:56.580853 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.580814    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Oct 31 12:49:56.835328 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.791048    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 12:49:56.835328 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.830288    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:56.838138 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.838103    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:49:56.956816 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:56.956778    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:57.014609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.009447    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:57.050582 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.050551    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:49:57.052527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.052488    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:57.131536 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.093526    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:57.131536 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.095925    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:49:57.176240 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.176207    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 12:49:57.315373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.315204    5663 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded"
Oct 31 12:49:57.334414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.332009    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:57.334414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.333041    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:49:57.334414 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.333087    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 12:49:57.345371 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.343766    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 31 12:49:57.420454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.392594    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:49:57.472548 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.472491    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:57.498728 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.498679    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:49:57.501367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.501306    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 12:49:57.503844 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.503817    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn" podUID="1d224a67-a771-46d2-ad68-c8970ca2dffa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:57.505638 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.505615    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:49:57.850540 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:57.850460    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:49:58.064400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:58.064369    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:49:58.064687 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:58.064671    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:49:58.064900 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:58.064885    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:49:58.103750 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:58.080863    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:49:58.192379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:58.155217    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:49:58.192379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:58.155289    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 12:49:58.247429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:58.245022    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:49:58.329413 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:58.325172    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:49:58.329413 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:58.325232    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 12:49:59.150372 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:59.150324    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:49:59.150789 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:59.150439    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 12:49:59.306437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:59.306077    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:49:59.306437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:49:59.306159    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 12:50:00.150388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:00.149894    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:50:00.150388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:00.149953    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 12:50:00.301381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:00.300749    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 31 12:50:00.301381 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:00.300807    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 31 12:50:01.163809 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:01.163770    5663 patch_prober.go:29] interesting pod/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 12:50:01.164476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:01.163835    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 12:50:01.421219 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:01.421121    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:50:01.421689 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:01.421652    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:50:01.422493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:01.422474    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:50:02.168752 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:02.168717    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:50:02.169718 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:02.169691    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:50:02.176670 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:02.176644    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:50:02.396980 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:50:02.396948    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"
Oct 31 12:50:22.479974 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:50:22.479934    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:50:52.478704 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:50:52.478625    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:51:22.489941 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:51:22.489891    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:51:52.477924 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:51:52.477889    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:52:22.478306 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:52:22.478274    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:52:52.483739 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:52:52.483699    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:52:54.565175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:52:54.565122    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"]
Oct 31 12:52:54.565751 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:52:54.565572    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerName="istio-proxy" containerID="cri-o://68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37" gracePeriod=30
Oct 31 12:52:59.924825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:52:59.924796    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:53:00.030929 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.030892    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-data\") pod \"1b55a494-6e14-4759-b8d7-53bf788aab78\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") "
Oct 31 12:53:00.031250 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.031205    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-credential-socket\") pod \"1b55a494-6e14-4759-b8d7-53bf788aab78\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") "
Oct 31 12:53:00.031603 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.031584    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-podinfo\") pod \"1b55a494-6e14-4759-b8d7-53bf788aab78\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") "
Oct 31 12:53:00.031869 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.031852    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b55a494-6e14-4759-b8d7-53bf788aab78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b55a494-6e14-4759-b8d7-53bf788aab78-istiod-ca-cert\") pod \"1b55a494-6e14-4759-b8d7-53bf788aab78\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") "
Oct 31 12:53:00.032035 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.032019    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-token\" (UniqueName: \"kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-token\") pod \"1b55a494-6e14-4759-b8d7-53bf788aab78\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") "
Oct 31 12:53:00.032293 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.032274    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-certs\") pod \"1b55a494-6e14-4759-b8d7-53bf788aab78\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") "
Oct 31 12:53:00.032717 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.032696    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-envoy\") pod \"1b55a494-6e14-4759-b8d7-53bf788aab78\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") "
Oct 31 12:53:00.032842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.032827    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-socket\") pod \"1b55a494-6e14-4759-b8d7-53bf788aab78\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") "
Oct 31 12:53:00.032961 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.032948    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-kube-api-access-n9nj8\" (UniqueName: \"kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-kube-api-access-n9nj8\") pod \"1b55a494-6e14-4759-b8d7-53bf788aab78\" (UID: \"1b55a494-6e14-4759-b8d7-53bf788aab78\") "
Oct 31 12:53:00.082779 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.071674    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-certs" pod "1b55a494-6e14-4759-b8d7-53bf788aab78" (UID: "1b55a494-6e14-4759-b8d7-53bf788aab78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.082779 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.076402    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-data" pod "1b55a494-6e14-4759-b8d7-53bf788aab78" (UID: "1b55a494-6e14-4759-b8d7-53bf788aab78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.082779 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.080489    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-credential-socket" pod "1b55a494-6e14-4759-b8d7-53bf788aab78" (UID: "1b55a494-6e14-4759-b8d7-53bf788aab78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.082779 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.080657    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b55a494-6e14-4759-b8d7-53bf788aab78-istiod-ca-cert" pod "1b55a494-6e14-4759-b8d7-53bf788aab78" (UID: "1b55a494-6e14-4759-b8d7-53bf788aab78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:00.082779 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.081498    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-socket" pod "1b55a494-6e14-4759-b8d7-53bf788aab78" (UID: "1b55a494-6e14-4759-b8d7-53bf788aab78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.135979 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.135945    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b55a494-6e14-4759-b8d7-53bf788aab78-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.135979 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.135982    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.136949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.135997    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.136949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.136011    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.136949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.136025    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.136949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.136503    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-envoy" pod "1b55a494-6e14-4759-b8d7-53bf788aab78" (UID: "1b55a494-6e14-4759-b8d7-53bf788aab78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:00.136949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.136608    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-podinfo" pod "1b55a494-6e14-4759-b8d7-53bf788aab78" (UID: "1b55a494-6e14-4759-b8d7-53bf788aab78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:00.137639 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.137587    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-token" pod "1b55a494-6e14-4759-b8d7-53bf788aab78" (UID: "1b55a494-6e14-4759-b8d7-53bf788aab78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:00.149087 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.149056    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"]
Oct 31 12:53:00.150280 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.150256    5663 cpu_manager.go:414] "containerMap: removing container" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerName="istio-proxy"
Oct 31 12:53:00.150280 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.150281    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerName="istio-proxy"
Oct 31 12:53:00.150489 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.150475    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerName="istio-proxy"
Oct 31 12:53:00.152451 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.152410    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-kube-api-access-n9nj8" pod "1b55a494-6e14-4759-b8d7-53bf788aab78" (UID: "1b55a494-6e14-4759-b8d7-53bf788aab78"). InnerVolumeSpecName "kube-api-access-n9nj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:00.156166 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.156108    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"]
Oct 31 12:53:00.156274 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.156221    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.164812 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.164788    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2a0ea4-serving-cert\""
Oct 31 12:53:00.237151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.237116    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-metrics-certs\") pod \"maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.237324 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.237171    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.237324 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.237208    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p5vh\" (UniqueName: \"kubernetes.io/projected/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-kube-api-access-6p5vh\") pod \"maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.237465 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.237448    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.237564 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.237554    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.237609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.237568    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9nj8\" (UniqueName: \"kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-kube-api-access-n9nj8\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.237609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.237577    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.237609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.237586    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b55a494-6e14-4759-b8d7-53bf788aab78-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:00.338902 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.338868    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-metrics-certs\") pod \"maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.338902 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.338910    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.339143 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.339042    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p5vh\" (UniqueName: \"kubernetes.io/projected/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-kube-api-access-6p5vh\") pod \"maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.339909 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.339877    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.340480 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.340439    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.341964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.341939    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-metrics-certs\") pod \"maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.342088 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.342008    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.363156 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.363124    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p5vh\" (UniqueName: \"kubernetes.io/projected/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-kube-api-access-6p5vh\") pod \"maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.473078 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.472986    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:00.620768 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.620731    5663 generic.go:356] "Generic (PLEG): container finished" podID="1b55a494-6e14-4759-b8d7-53bf788aab78" containerID="68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37" exitCode=0
Oct 31 12:53:00.620894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.620790    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" event={"ID":"1b55a494-6e14-4759-b8d7-53bf788aab78","Type":"ContainerDied","Data":"68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37"}
Oct 31 12:53:00.620894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.620841    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82" event={"ID":"1b55a494-6e14-4759-b8d7-53bf788aab78","Type":"ContainerDied","Data":"b3f9a6a4c40b8dc08e708f9ad3b06a1f66d19342d02bfd74fd46a4c952cccb87"}
Oct 31 12:53:00.620894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.620871    5663 scope.go:122] "RemoveContainer" containerID="68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37"
Oct 31 12:53:00.621065 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.621039    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"
Oct 31 12:53:00.638700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.638675    5663 scope.go:122] "RemoveContainer" containerID="68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37"
Oct 31 12:53:00.639008 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:53:00.638986    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37\": container with ID starting with 68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37 not found: ID does not exist" containerID="68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37"
Oct 31 12:53:00.639109 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.639014    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37"} err="failed to get container status \"68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37\": rpc error: code = NotFound desc = could not find container \"68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37\": container with ID starting with 68b00e117bfb62c83d16e4eba9559055218c32ce6a58220afbd09677a749ac37 not found: ID does not exist"
Oct 31 12:53:00.657817 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.657794    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"]
Oct 31 12:53:00.663382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.663342    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-wgg82"]
Oct 31 12:53:00.778854 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:00.778825    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"]
Oct 31 12:53:00.779488 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:53:00.779460    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b18aea_9ac2_44bb_8c98_6de55fc5e1eb.slice/crio-8381a61078c320e0134e6437307a7809377d9ebb73f784d00d556bb849b20da4 WatchSource:0}: Error finding container 8381a61078c320e0134e6437307a7809377d9ebb73f784d00d556bb849b20da4: Status 404 returned error can't find the container with id 8381a61078c320e0134e6437307a7809377d9ebb73f784d00d556bb849b20da4
Oct 31 12:53:01.549045 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:01.548883    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b55a494-6e14-4759-b8d7-53bf788aab78" path="/var/lib/kubelet/pods/1b55a494-6e14-4759-b8d7-53bf788aab78/volumes"
Oct 31 12:53:01.635039 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:01.635006    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x" event={"ID":"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb","Type":"ContainerStarted","Data":"aaa746df2bcca24f66c38b6454ddbd8bc9805ffee7e58fa138f9929514fc6a84"}
Oct 31 12:53:01.635252 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:01.635058    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x" event={"ID":"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb","Type":"ContainerStarted","Data":"8381a61078c320e0134e6437307a7809377d9ebb73f784d00d556bb849b20da4"}
Oct 31 12:53:01.635252 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:01.635082    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:01.663411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:01.663343    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x" podStartSLOduration=166.206159686 podStartE2EDuration="2m46.663328797s" totalImagesPullingTime="457.169111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:01.661863361 +0000 UTC m=+1506.402289574" watchObservedRunningTime="2026-10-31 12:53:01.663328797 +0000 UTC m=+1506.403755054"
Oct 31 12:53:03.975485 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:03.975446    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"]
Oct 31 12:53:04.019060 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:04.019026    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"]
Oct 31 12:53:07.648845 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:07.648816    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:08.990382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:08.978410    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l" podUID="1c8c1faf-d346-4596-9e50-ab782ae40b2f" containerName="payload-processing" containerID="cri-o://ef76890a0d8143f09f02e643b32eb35b37fa8a9ff8b0ce30a3b49cb84fa005ac" gracePeriod=25
Oct 31 12:53:09.047383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:09.025401    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn" podUID="41979d00-c01e-4bfa-953f-f70f7fc4d22b" containerName="payload-pre-processing" containerID="cri-o://d6b16fb4e4a6512c1c429b6149bed7897ea6101ad2646e20638e4766b727fbef" gracePeriod=25
Oct 31 12:53:09.855433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:09.730886    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:09.855433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:09.842480    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"]
Oct 31 12:53:09.855433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:09.842548    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"]
Oct 31 12:53:09.855433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:09.842744    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" podUID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" containerName="istio-proxy" containerID="cri-o://455b83d46a21d1112ed78e7f574b4c9eb36231c0682284939f1b034e2b637a99" gracePeriod=30
Oct 31 12:53:09.855433 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:09.843047    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy" containerID="cri-o://3950c3ffd9b4906c4612d07c7b7758690ae38a84edf1b979cef879cc2bba0ef2" gracePeriod=30
Oct 31 12:53:09.868689 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:09.867323    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"]
Oct 31 12:53:09.870078 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:09.870043    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerName="istio-proxy" containerID="cri-o://74475ac74a756ceaad112d68e691f6cfa356dca69a75f9978308ac3d0fffa7e4" gracePeriod=30
Oct 31 12:53:09.883149 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:09.882807    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"]
Oct 31 12:53:09.883149 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:09.883029    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55" podUID="1778cc89-0d95-46b0-b633-66a43483dc2d" containerName="istio-proxy" containerID="cri-o://b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329" gracePeriod=30
Oct 31 12:53:10.262404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.256747    5663 generic.go:356] "Generic (PLEG): container finished" podID="41979d00-c01e-4bfa-953f-f70f7fc4d22b" containerID="d6b16fb4e4a6512c1c429b6149bed7897ea6101ad2646e20638e4766b727fbef" exitCode=0
Oct 31 12:53:10.262404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.256880    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn" event={"ID":"41979d00-c01e-4bfa-953f-f70f7fc4d22b","Type":"ContainerDied","Data":"d6b16fb4e4a6512c1c429b6149bed7897ea6101ad2646e20638e4766b727fbef"}
Oct 31 12:53:10.262404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.260750    5663 generic.go:356] "Generic (PLEG): container finished" podID="1c8c1faf-d346-4596-9e50-ab782ae40b2f" containerID="ef76890a0d8143f09f02e643b32eb35b37fa8a9ff8b0ce30a3b49cb84fa005ac" exitCode=0
Oct 31 12:53:10.262404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.260827    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l" event={"ID":"1c8c1faf-d346-4596-9e50-ab782ae40b2f","Type":"ContainerDied","Data":"ef76890a0d8143f09f02e643b32eb35b37fa8a9ff8b0ce30a3b49cb84fa005ac"}
Oct 31 12:53:10.309938 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.309906    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"
Oct 31 12:53:10.345563 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.345520    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"]
Oct 31 12:53:10.346718 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.346691    5663 cpu_manager.go:414] "containerMap: removing container" podUID="1c8c1faf-d346-4596-9e50-ab782ae40b2f" containerName="payload-processing"
Oct 31 12:53:10.346718 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.346716    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8c1faf-d346-4596-9e50-ab782ae40b2f" containerName="payload-processing"
Oct 31 12:53:10.347041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.346910    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8c1faf-d346-4596-9e50-ab782ae40b2f" containerName="payload-processing"
Oct 31 12:53:10.366708 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.366677    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"]
Oct 31 12:53:10.366837 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.366816    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"
Oct 31 12:53:10.383378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.379821    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d176a5\""
Oct 31 12:53:10.383378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.380181    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d176a5-dockercfg-4b7ln\""
Oct 31 12:53:10.396334 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.396268    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"
Oct 31 12:53:10.424400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.424366    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c8c1faf-d346-4596-9e50-ab782ae40b2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c1faf-d346-4596-9e50-ab782ae40b2f-plugins-config-volume\") pod \"1c8c1faf-d346-4596-9e50-ab782ae40b2f\" (UID: \"1c8c1faf-d346-4596-9e50-ab782ae40b2f\") "
Oct 31 12:53:10.424559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.424549    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8c1faf-d346-4596-9e50-ab782ae40b2f-kube-api-access-bz2hf\" (UniqueName: \"kubernetes.io/projected/1c8c1faf-d346-4596-9e50-ab782ae40b2f-kube-api-access-bz2hf\") pod \"1c8c1faf-d346-4596-9e50-ab782ae40b2f\" (UID: \"1c8c1faf-d346-4596-9e50-ab782ae40b2f\") "
Oct 31 12:53:10.424747 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.424722    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8c1faf-d346-4596-9e50-ab782ae40b2f-plugins-config-volume" pod "1c8c1faf-d346-4596-9e50-ab782ae40b2f" (UID: "1c8c1faf-d346-4596-9e50-ab782ae40b2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:10.426252 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.425406    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8c1faf-d346-4596-9e50-ab782ae40b2f-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:10.427121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.427094    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8c1faf-d346-4596-9e50-ab782ae40b2f-kube-api-access-bz2hf" pod "1c8c1faf-d346-4596-9e50-ab782ae40b2f" (UID: "1c8c1faf-d346-4596-9e50-ab782ae40b2f"). InnerVolumeSpecName "kube-api-access-bz2hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:10.526450 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.526415    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41979d00-c01e-4bfa-953f-f70f7fc4d22b-kube-api-access-5xmph\" (UniqueName: \"kubernetes.io/projected/41979d00-c01e-4bfa-953f-f70f7fc4d22b-kube-api-access-5xmph\") pod \"41979d00-c01e-4bfa-953f-f70f7fc4d22b\" (UID: \"41979d00-c01e-4bfa-953f-f70f7fc4d22b\") "
Oct 31 12:53:10.526621 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.526478    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41979d00-c01e-4bfa-953f-f70f7fc4d22b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41979d00-c01e-4bfa-953f-f70f7fc4d22b-plugins-config-volume\") pod \"41979d00-c01e-4bfa-953f-f70f7fc4d22b\" (UID: \"41979d00-c01e-4bfa-953f-f70f7fc4d22b\") "
Oct 31 12:53:10.526920 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.526893    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41979d00-c01e-4bfa-953f-f70f7fc4d22b-plugins-config-volume" pod "41979d00-c01e-4bfa-953f-f70f7fc4d22b" (UID: "41979d00-c01e-4bfa-953f-f70f7fc4d22b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:10.527403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.527374    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj\" (UID: \"1ceb5e4b-0314-4f7e-bd59-ec883e6db112\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"
Oct 31 12:53:10.527527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.527498    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf9v6\" (UniqueName: \"kubernetes.io/projected/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-kube-api-access-mf9v6\") pod \"payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj\" (UID: \"1ceb5e4b-0314-4f7e-bd59-ec883e6db112\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"
Oct 31 12:53:10.527620 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.527606    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz2hf\" (UniqueName: \"kubernetes.io/projected/1c8c1faf-d346-4596-9e50-ab782ae40b2f-kube-api-access-bz2hf\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:10.527685 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.527626    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41979d00-c01e-4bfa-953f-f70f7fc4d22b-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:10.528669 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.528646    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41979d00-c01e-4bfa-953f-f70f7fc4d22b-kube-api-access-5xmph" pod "41979d00-c01e-4bfa-953f-f70f7fc4d22b" (UID: "41979d00-c01e-4bfa-953f-f70f7fc4d22b"). InnerVolumeSpecName "kube-api-access-5xmph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:10.629571 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.629537    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj\" (UID: \"1ceb5e4b-0314-4f7e-bd59-ec883e6db112\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"
Oct 31 12:53:10.629738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.629592    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf9v6\" (UniqueName: \"kubernetes.io/projected/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-kube-api-access-mf9v6\") pod \"payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj\" (UID: \"1ceb5e4b-0314-4f7e-bd59-ec883e6db112\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"
Oct 31 12:53:10.629790 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.629738    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xmph\" (UniqueName: \"kubernetes.io/projected/41979d00-c01e-4bfa-953f-f70f7fc4d22b-kube-api-access-5xmph\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:10.630331 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.630310    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj\" (UID: \"1ceb5e4b-0314-4f7e-bd59-ec883e6db112\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"
Oct 31 12:53:10.650259 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.650200    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf9v6\" (UniqueName: \"kubernetes.io/projected/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-kube-api-access-mf9v6\") pod \"payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj\" (UID: \"1ceb5e4b-0314-4f7e-bd59-ec883e6db112\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"
Oct 31 12:53:10.710444 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:10.710416    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"
Oct 31 12:53:11.103712 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.103686    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"]
Oct 31 12:53:11.104544 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:53:11.104520    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ceb5e4b_0314_4f7e_bd59_ec883e6db112.slice/crio-789583f193ff66eff5ad33ee574d73828f888123ff51af3d488dcbbbabf9fc10 WatchSource:0}: Error finding container 789583f193ff66eff5ad33ee574d73828f888123ff51af3d488dcbbbabf9fc10: Status 404 returned error can't find the container with id 789583f193ff66eff5ad33ee574d73828f888123ff51af3d488dcbbbabf9fc10
Oct 31 12:53:11.270915 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.270877    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj" event={"ID":"1ceb5e4b-0314-4f7e-bd59-ec883e6db112","Type":"ContainerStarted","Data":"2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92"}
Oct 31 12:53:11.271398 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.270951    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"
Oct 31 12:53:11.271398 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.270976    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj" event={"ID":"1ceb5e4b-0314-4f7e-bd59-ec883e6db112","Type":"ContainerStarted","Data":"789583f193ff66eff5ad33ee574d73828f888123ff51af3d488dcbbbabf9fc10"}
Oct 31 12:53:11.272463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.272432    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"
Oct 31 12:53:11.272548 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.272467    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn" event={"ID":"41979d00-c01e-4bfa-953f-f70f7fc4d22b","Type":"ContainerDied","Data":"7003b111687d89d483ce33109ce36f5eccbebb4b74e4b61cefbecfeab5ee1e15"}
Oct 31 12:53:11.272548 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.272533    5663 scope.go:122] "RemoveContainer" containerID="d6b16fb4e4a6512c1c429b6149bed7897ea6101ad2646e20638e4766b727fbef"
Oct 31 12:53:11.274121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.274105    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"
Oct 31 12:53:11.274235 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.274118    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l" event={"ID":"1c8c1faf-d346-4596-9e50-ab782ae40b2f","Type":"ContainerDied","Data":"f410b101ce9a6804288058aa39ef2c92e3992a354ebb38aed833bd1085ed0454"}
Oct 31 12:53:11.293836 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.293820    5663 scope.go:122] "RemoveContainer" containerID="ef76890a0d8143f09f02e643b32eb35b37fa8a9ff8b0ce30a3b49cb84fa005ac"
Oct 31 12:53:11.320396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.320335    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj" podStartSLOduration=3.32032257 podStartE2EDuration="3.32032257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:11.318044587 +0000 UTC m=+1516.058470803" watchObservedRunningTime="2026-10-31 12:53:11.32032257 +0000 UTC m=+1516.060748812"
Oct 31 12:53:11.354684 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.354618    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"]
Oct 31 12:53:11.365326 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.365302    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4e8eb9-7ddf9fdc55-cqjzn"]
Oct 31 12:53:11.385046 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.385016    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"]
Oct 31 12:53:11.393453 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.393423    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4e8eb9-7d7787f9ff-lwc2l"]
Oct 31 12:53:11.494578 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.487557    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8c1faf-d346-4596-9e50-ab782ae40b2f" path="/var/lib/kubelet/pods/1c8c1faf-d346-4596-9e50-ab782ae40b2f/volumes"
Oct 31 12:53:11.494578 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.488682    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41979d00-c01e-4bfa-953f-f70f7fc4d22b" path="/var/lib/kubelet/pods/41979d00-c01e-4bfa-953f-f70f7fc4d22b/volumes"
Oct 31 12:53:11.979839 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.979802    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"]
Oct 31 12:53:11.980051 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:11.980022    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz" podUID="8729e399-98e5-4087-bb60-2479bd32e646" containerName="maas-api" containerID="cri-o://5c05a69412f2d20505069be6bbb6fb1f9a78d68f18219fe4f80fa3fa85ce755f" gracePeriod=30
Oct 31 12:53:12.522658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.522183    5663 generic.go:356] "Generic (PLEG): container finished" podID="8729e399-98e5-4087-bb60-2479bd32e646" containerID="5c05a69412f2d20505069be6bbb6fb1f9a78d68f18219fe4f80fa3fa85ce755f" exitCode=0
Oct 31 12:53:12.522658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.522318    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz" event={"ID":"8729e399-98e5-4087-bb60-2479bd32e646","Type":"ContainerDied","Data":"5c05a69412f2d20505069be6bbb6fb1f9a78d68f18219fe4f80fa3fa85ce755f"}
Oct 31 12:53:12.772603 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.772579    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:53:12.821989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.821939    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"]
Oct 31 12:53:12.823236 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.823204    5663 cpu_manager.go:414] "containerMap: removing container" podUID="8729e399-98e5-4087-bb60-2479bd32e646" containerName="maas-api"
Oct 31 12:53:12.823236 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.823231    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8729e399-98e5-4087-bb60-2479bd32e646" containerName="maas-api"
Oct 31 12:53:12.823528 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.823258    5663 cpu_manager.go:414] "containerMap: removing container" podUID="41979d00-c01e-4bfa-953f-f70f7fc4d22b" containerName="payload-pre-processing"
Oct 31 12:53:12.823528 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.823267    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41979d00-c01e-4bfa-953f-f70f7fc4d22b" containerName="payload-pre-processing"
Oct 31 12:53:12.823528 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.823457    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="8729e399-98e5-4087-bb60-2479bd32e646" containerName="maas-api"
Oct 31 12:53:12.823528 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.823477    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="41979d00-c01e-4bfa-953f-f70f7fc4d22b" containerName="payload-pre-processing"
Oct 31 12:53:12.827944 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.827899    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:12.836931 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.836901    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e98df8-serving-cert\""
Oct 31 12:53:12.847983 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.845135    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"]
Oct 31 12:53:12.893572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.893541    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8729e399-98e5-4087-bb60-2479bd32e646-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8729e399-98e5-4087-bb60-2479bd32e646-openshift-service-ca\") pod \"8729e399-98e5-4087-bb60-2479bd32e646\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") "
Oct 31 12:53:12.893770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.893724    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8729e399-98e5-4087-bb60-2479bd32e646-kube-api-access-5nl48\" (UniqueName: \"kubernetes.io/projected/8729e399-98e5-4087-bb60-2479bd32e646-kube-api-access-5nl48\") pod \"8729e399-98e5-4087-bb60-2479bd32e646\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") "
Oct 31 12:53:12.893898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.893874    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8729e399-98e5-4087-bb60-2479bd32e646-openshift-service-ca" pod "8729e399-98e5-4087-bb60-2479bd32e646" (UID: "8729e399-98e5-4087-bb60-2479bd32e646"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:12.894026 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.894010    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-maas-api-tls\") pod \"8729e399-98e5-4087-bb60-2479bd32e646\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") "
Oct 31 12:53:12.894083 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.894070    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-metrics-certs\") pod \"8729e399-98e5-4087-bb60-2479bd32e646\" (UID: \"8729e399-98e5-4087-bb60-2479bd32e646\") "
Oct 31 12:53:12.894425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.894405    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71690981-0f09-4b44-a93f-ce307e7ff025-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:12.894545 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.894526    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-metrics-certs\") pod \"maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:12.894796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.894779    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-maas-api-tls\") pod \"maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:12.895053 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.895029    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbcpb\" (UniqueName: \"kubernetes.io/projected/71690981-0f09-4b44-a93f-ce307e7ff025-kube-api-access-gbcpb\") pod \"maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:12.895202 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.895187    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8729e399-98e5-4087-bb60-2479bd32e646-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:12.895912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.895883    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8729e399-98e5-4087-bb60-2479bd32e646-kube-api-access-5nl48" pod "8729e399-98e5-4087-bb60-2479bd32e646" (UID: "8729e399-98e5-4087-bb60-2479bd32e646"). InnerVolumeSpecName "kube-api-access-5nl48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:12.896083 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.896059    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-maas-api-tls" pod "8729e399-98e5-4087-bb60-2479bd32e646" (UID: "8729e399-98e5-4087-bb60-2479bd32e646"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:53:12.896205 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:12.896189    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-metrics-certs" pod "8729e399-98e5-4087-bb60-2479bd32e646" (UID: "8729e399-98e5-4087-bb60-2479bd32e646"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:53:13.001727 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.001682    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-metrics-certs\") pod \"maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:13.001998 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.001968    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-maas-api-tls\") pod \"maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:13.002256 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.002158    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbcpb\" (UniqueName: \"kubernetes.io/projected/71690981-0f09-4b44-a93f-ce307e7ff025-kube-api-access-gbcpb\") pod \"maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:13.002256 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.002225    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71690981-0f09-4b44-a93f-ce307e7ff025-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:13.003028 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.003000    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:13.003151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.003041    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nl48\" (UniqueName: \"kubernetes.io/projected/8729e399-98e5-4087-bb60-2479bd32e646-kube-api-access-5nl48\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:13.003151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.003083    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8729e399-98e5-4087-bb60-2479bd32e646-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:13.011425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.010546    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71690981-0f09-4b44-a93f-ce307e7ff025-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:13.013493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.013461    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-maas-api-tls\") pod \"maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:13.015791 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.015764    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-metrics-certs\") pod \"maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:13.022840 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.022818    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbcpb\" (UniqueName: \"kubernetes.io/projected/71690981-0f09-4b44-a93f-ce307e7ff025-kube-api-access-gbcpb\") pod \"maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:13.151957 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.151884    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:13.444284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.444257    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"]
Oct 31 12:53:13.444696 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:53:13.444670    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71690981_0f09_4b44_a93f_ce307e7ff025.slice/crio-d2a482db188b65bbf81bb217546417d090746b746721e4cc5adf2b23bda6af26 WatchSource:0}: Error finding container d2a482db188b65bbf81bb217546417d090746b746721e4cc5adf2b23bda6af26: Status 404 returned error can't find the container with id d2a482db188b65bbf81bb217546417d090746b746721e4cc5adf2b23bda6af26
Oct 31 12:53:13.554593 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.554561    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"
Oct 31 12:53:13.555043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.554585    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz" event={"ID":"8729e399-98e5-4087-bb60-2479bd32e646","Type":"ContainerDied","Data":"eeb85889e6ad72458bab8c4862ace00cf179c39cf61020ccb0d287685c3ca5d6"}
Oct 31 12:53:13.555043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.554653    5663 scope.go:122] "RemoveContainer" containerID="5c05a69412f2d20505069be6bbb6fb1f9a78d68f18219fe4f80fa3fa85ce755f"
Oct 31 12:53:13.556153 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.556126    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq" event={"ID":"71690981-0f09-4b44-a93f-ce307e7ff025","Type":"ContainerStarted","Data":"d2a482db188b65bbf81bb217546417d090746b746721e4cc5adf2b23bda6af26"}
Oct 31 12:53:13.579006 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.578983    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"]
Oct 31 12:53:13.601731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:13.601707    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4e8eb9-6cc7b96d85-76chz"]
Oct 31 12:53:14.565368 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:14.565318    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq" event={"ID":"71690981-0f09-4b44-a93f-ce307e7ff025","Type":"ContainerStarted","Data":"24ebd9a24d794ceef879837b7fd31913e5770c89160a4a8c86ddd33bde45a322"}
Oct 31 12:53:14.565861 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:14.565396    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:14.592718 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:14.592673    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq" podStartSLOduration=244.024931002 podStartE2EDuration="4m4.592659926s" totalImagesPullingTime="567.728924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:14.588164941 +0000 UTC m=+1519.328591156" watchObservedRunningTime="2026-10-31 12:53:14.592659926 +0000 UTC m=+1519.333086147"
Oct 31 12:53:14.978227 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:53:14.977211    5663 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode183677e_396a_41f6_aa42_1184c82d520b.slice/crio-3950c3ffd9b4906c4612d07c7b7758690ae38a84edf1b979cef879cc2bba0ef2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1778cc89_0d95_46b0_b633_66a43483dc2d.slice/crio-b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 12:53:15.182325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.170804    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 12:53:15.182325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.170864    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55" podUID="1778cc89-0d95-46b0-b633-66a43483dc2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 12:53:15.204074 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.204043    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"]
Oct 31 12:53:15.210753 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.210726    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"
Oct 31 12:53:15.223951 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.223476    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d086f3-dockercfg-rv558\""
Oct 31 12:53:15.223951 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.223572    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d086f3\""
Oct 31 12:53:15.239604 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.239555    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"]
Oct 31 12:53:15.251399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.251115    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5phd\" (UniqueName: \"kubernetes.io/projected/71f3a982-4cac-4228-b891-dca75a989765-kube-api-access-z5phd\") pod \"payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7\" (UID: \"71f3a982-4cac-4228-b891-dca75a989765\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"
Oct 31 12:53:15.251399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.251294    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f3a982-4cac-4228-b891-dca75a989765-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7\" (UID: \"71f3a982-4cac-4228-b891-dca75a989765\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"
Oct 31 12:53:15.364376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.358857    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5phd\" (UniqueName: \"kubernetes.io/projected/71f3a982-4cac-4228-b891-dca75a989765-kube-api-access-z5phd\") pod \"payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7\" (UID: \"71f3a982-4cac-4228-b891-dca75a989765\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"
Oct 31 12:53:15.364376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.359053    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f3a982-4cac-4228-b891-dca75a989765-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7\" (UID: \"71f3a982-4cac-4228-b891-dca75a989765\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"
Oct 31 12:53:15.364376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.360154    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f3a982-4cac-4228-b891-dca75a989765-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7\" (UID: \"71f3a982-4cac-4228-b891-dca75a989765\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"
Oct 31 12:53:15.566522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.433155    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5phd\" (UniqueName: \"kubernetes.io/projected/71f3a982-4cac-4228-b891-dca75a989765-kube-api-access-z5phd\") pod \"payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7\" (UID: \"71f3a982-4cac-4228-b891-dca75a989765\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"
Oct 31 12:53:15.620380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.573681    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"
Oct 31 12:53:15.721429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.682234    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 12:53:15.721429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.682301    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" podUID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 12:53:15.773975 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.771887    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:53:15.778221 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.778169    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8729e399-98e5-4087-bb60-2479bd32e646" path="/var/lib/kubelet/pods/8729e399-98e5-4087-bb60-2479bd32e646/volumes"
Oct 31 12:53:15.790944 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.790915    5663 generic.go:356] "Generic (PLEG): container finished" podID="e183677e-396a-41f6-aa42-1184c82d520b" containerID="3950c3ffd9b4906c4612d07c7b7758690ae38a84edf1b979cef879cc2bba0ef2" exitCode=0
Oct 31 12:53:15.791225 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.791202    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" event={"ID":"e183677e-396a-41f6-aa42-1184c82d520b","Type":"ContainerDied","Data":"3950c3ffd9b4906c4612d07c7b7758690ae38a84edf1b979cef879cc2bba0ef2"}
Oct 31 12:53:15.793126 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.793100    5663 generic.go:356] "Generic (PLEG): container finished" podID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerID="74475ac74a756ceaad112d68e691f6cfa356dca69a75f9978308ac3d0fffa7e4" exitCode=0
Oct 31 12:53:15.793342 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.793323    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" event={"ID":"a78f4e51-551c-43f8-a199-01ebd2c11e1a","Type":"ContainerDied","Data":"74475ac74a756ceaad112d68e691f6cfa356dca69a75f9978308ac3d0fffa7e4"}
Oct 31 12:53:15.794284 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.793744    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:53:15.796201 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.795985    5663 generic.go:356] "Generic (PLEG): container finished" podID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" containerID="455b83d46a21d1112ed78e7f574b4c9eb36231c0682284939f1b034e2b637a99" exitCode=0
Oct 31 12:53:15.796201 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.796174    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" event={"ID":"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9","Type":"ContainerDied","Data":"455b83d46a21d1112ed78e7f574b4c9eb36231c0682284939f1b034e2b637a99"}
Oct 31 12:53:15.800018 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.798443    5663 generic.go:356] "Generic (PLEG): container finished" podID="1778cc89-0d95-46b0-b633-66a43483dc2d" containerID="b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329" exitCode=0
Oct 31 12:53:15.800018 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.799503    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"
Oct 31 12:53:15.800018 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.799643    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55" event={"ID":"1778cc89-0d95-46b0-b633-66a43483dc2d","Type":"ContainerDied","Data":"b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329"}
Oct 31 12:53:15.800018 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.799679    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55" event={"ID":"1778cc89-0d95-46b0-b633-66a43483dc2d","Type":"ContainerDied","Data":"b91d3cc010b70bad169c3be8055e77e2c93bf7a098525a8e65caa94109165b31"}
Oct 31 12:53:15.800018 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.799706    5663 scope.go:122] "RemoveContainer" containerID="b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329"
Oct 31 12:53:15.825008 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.824683    5663 scope.go:122] "RemoveContainer" containerID="b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329"
Oct 31 12:53:15.825450 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:53:15.825250    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329\": container with ID starting with b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329 not found: ID does not exist" containerID="b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329"
Oct 31 12:53:15.825450 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.825290    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329"} err="failed to get container status \"b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329\": rpc error: code = NotFound desc = could not find container \"b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329\": container with ID starting with b4301385a79edbe43152f4869f856d5e2b6924173364ee6a92b8483c1d1c5329 not found: ID does not exist"
Oct 31 12:53:15.842033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.841020    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-envoy\") pod \"1778cc89-0d95-46b0-b633-66a43483dc2d\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") "
Oct 31 12:53:15.842033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.841264    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1778cc89-0d95-46b0-b633-66a43483dc2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1778cc89-0d95-46b0-b633-66a43483dc2d-istiod-ca-cert\") pod \"1778cc89-0d95-46b0-b633-66a43483dc2d\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") "
Oct 31 12:53:15.842033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.841369    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-token\" (UniqueName: \"kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-token\") pod \"1778cc89-0d95-46b0-b633-66a43483dc2d\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") "
Oct 31 12:53:15.842033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.841421    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-certs\") pod \"1778cc89-0d95-46b0-b633-66a43483dc2d\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") "
Oct 31 12:53:15.842033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.841496    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-socket\") pod \"1778cc89-0d95-46b0-b633-66a43483dc2d\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") "
Oct 31 12:53:15.842033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.841537    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-kube-api-access-x78r2\" (UniqueName: \"kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-kube-api-access-x78r2\") pod \"1778cc89-0d95-46b0-b633-66a43483dc2d\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") "
Oct 31 12:53:15.842873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.842661    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-podinfo\") pod \"1778cc89-0d95-46b0-b633-66a43483dc2d\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") "
Oct 31 12:53:15.842873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.842712    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-credential-socket\") pod \"1778cc89-0d95-46b0-b633-66a43483dc2d\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") "
Oct 31 12:53:15.842873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.842761    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-data\") pod \"1778cc89-0d95-46b0-b633-66a43483dc2d\" (UID: \"1778cc89-0d95-46b0-b633-66a43483dc2d\") "
Oct 31 12:53:15.843723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.843652    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-credential-socket" pod "1778cc89-0d95-46b0-b633-66a43483dc2d" (UID: "1778cc89-0d95-46b0-b633-66a43483dc2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:15.844233 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.844056    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-data" pod "1778cc89-0d95-46b0-b633-66a43483dc2d" (UID: "1778cc89-0d95-46b0-b633-66a43483dc2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:15.844233 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.844156    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1778cc89-0d95-46b0-b633-66a43483dc2d-istiod-ca-cert" pod "1778cc89-0d95-46b0-b633-66a43483dc2d" (UID: "1778cc89-0d95-46b0-b633-66a43483dc2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:15.844485 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.844439    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-certs" pod "1778cc89-0d95-46b0-b633-66a43483dc2d" (UID: "1778cc89-0d95-46b0-b633-66a43483dc2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:15.844579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.844490    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-socket" pod "1778cc89-0d95-46b0-b633-66a43483dc2d" (UID: "1778cc89-0d95-46b0-b633-66a43483dc2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:15.846121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.844805    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1778cc89-0d95-46b0-b633-66a43483dc2d-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:15.846121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.844833    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:15.846121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.844848    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:15.846121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.844863    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:15.846121 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.844882    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:15.850619 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.847548    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-podinfo" pod "1778cc89-0d95-46b0-b633-66a43483dc2d" (UID: "1778cc89-0d95-46b0-b633-66a43483dc2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:15.850619 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.848533    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-token" pod "1778cc89-0d95-46b0-b633-66a43483dc2d" (UID: "1778cc89-0d95-46b0-b633-66a43483dc2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:15.850619 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.850544    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-kube-api-access-x78r2" pod "1778cc89-0d95-46b0-b633-66a43483dc2d" (UID: "1778cc89-0d95-46b0-b633-66a43483dc2d"). InnerVolumeSpecName "kube-api-access-x78r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:15.850619 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.850580    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-envoy" pod "1778cc89-0d95-46b0-b633-66a43483dc2d" (UID: "1778cc89-0d95-46b0-b633-66a43483dc2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:15.946257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.946226    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-kube-api-access-kst48\" (UniqueName: \"kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-kube-api-access-kst48\") pod \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") "
Oct 31 12:53:15.946440 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.946272    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istiod-ca-cert\") pod \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") "
Oct 31 12:53:15.946440 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.946338    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-token\" (UniqueName: \"kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-token\") pod \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") "
Oct 31 12:53:15.946440 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.946414    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-envoy\") pod \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") "
Oct 31 12:53:15.946663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.946478    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-podinfo\") pod \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") "
Oct 31 12:53:15.946663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.946547    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-data\") pod \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") "
Oct 31 12:53:15.946663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.946610    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-credential-socket\") pod \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") "
Oct 31 12:53:15.946847 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.946742    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-certs\") pod \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") "
Oct 31 12:53:15.946920 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.946879    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istiod-ca-cert" pod "a78f4e51-551c-43f8-a199-01ebd2c11e1a" (UID: "a78f4e51-551c-43f8-a199-01ebd2c11e1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:15.946970 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.946958    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-socket\") pod \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\" (UID: \"a78f4e51-551c-43f8-a199-01ebd2c11e1a\") "
Oct 31 12:53:15.947426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.947377    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-credential-socket" pod "a78f4e51-551c-43f8-a199-01ebd2c11e1a" (UID: "a78f4e51-551c-43f8-a199-01ebd2c11e1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:15.947599 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.947577    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-socket" pod "a78f4e51-551c-43f8-a199-01ebd2c11e1a" (UID: "a78f4e51-551c-43f8-a199-01ebd2c11e1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:15.948145 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.948119    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:15.948145 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.948132    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-certs" pod "a78f4e51-551c-43f8-a199-01ebd2c11e1a" (UID: "a78f4e51-551c-43f8-a199-01ebd2c11e1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:15.948337 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.948148    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:15.948337 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.948174    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:15.948337 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.948189    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:15.948337 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.948204    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:15.948337 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.948217    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:15.948337 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.948232    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x78r2\" (UniqueName: \"kubernetes.io/projected/1778cc89-0d95-46b0-b633-66a43483dc2d-kube-api-access-x78r2\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:15.948791 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.948684    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-data" pod "a78f4e51-551c-43f8-a199-01ebd2c11e1a" (UID: "a78f4e51-551c-43f8-a199-01ebd2c11e1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:15.955222 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.953448    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-envoy" pod "a78f4e51-551c-43f8-a199-01ebd2c11e1a" (UID: "a78f4e51-551c-43f8-a199-01ebd2c11e1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:15.957194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.957171    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-podinfo" pod "a78f4e51-551c-43f8-a199-01ebd2c11e1a" (UID: "a78f4e51-551c-43f8-a199-01ebd2c11e1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:15.957324 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.957204    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-token" pod "a78f4e51-551c-43f8-a199-01ebd2c11e1a" (UID: "a78f4e51-551c-43f8-a199-01ebd2c11e1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:15.959160 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:15.959130    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-kube-api-access-kst48" pod "a78f4e51-551c-43f8-a199-01ebd2c11e1a" (UID: "a78f4e51-551c-43f8-a199-01ebd2c11e1a"). InnerVolumeSpecName "kube-api-access-kst48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:16.025842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.025813    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:53:16.041933 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:53:16.041898    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f3a982_4cac_4228_b891_dca75a989765.slice/crio-9999e7dedb9938708ba7c34925d52101b22a0f7b231023a8133ccafb6781d670 WatchSource:0}: Error finding container 9999e7dedb9938708ba7c34925d52101b22a0f7b231023a8133ccafb6781d670: Status 404 returned error can't find the container with id 9999e7dedb9938708ba7c34925d52101b22a0f7b231023a8133ccafb6781d670
Oct 31 12:53:16.043343 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.043311    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"]
Oct 31 12:53:16.057764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.052763    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.057764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.052790    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.057764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.052808    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.057764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.052822    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.057764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.052838    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a78f4e51-551c-43f8-a199-01ebd2c11e1a-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.057764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.052853    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kst48\" (UniqueName: \"kubernetes.io/projected/a78f4e51-551c-43f8-a199-01ebd2c11e1a-kube-api-access-kst48\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.079577 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.079511    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:53:16.139847 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.139777    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"]
Oct 31 12:53:16.143846 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.143817    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-knz55"]
Oct 31 12:53:16.153453 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.153422    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e183677e-396a-41f6-aa42-1184c82d520b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e183677e-396a-41f6-aa42-1184c82d520b-istio-podinfo\") pod \"e183677e-396a-41f6-aa42-1184c82d520b\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") "
Oct 31 12:53:16.153587 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.153480    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-certs\") pod \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") "
Oct 31 12:53:16.153587 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.153548    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-credential-socket\") pod \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") "
Oct 31 12:53:16.153696 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.153596    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-podinfo\") pod \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") "
Oct 31 12:53:16.153696 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.153623    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-envoy\") pod \"e183677e-396a-41f6-aa42-1184c82d520b\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") "
Oct 31 12:53:16.153696 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.153645    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-istio-token\" (UniqueName: \"kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-istio-token\") pod \"e183677e-396a-41f6-aa42-1184c82d520b\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") "
Oct 31 12:53:16.153696 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.153673    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-kube-api-access-db6r5\" (UniqueName: \"kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-kube-api-access-db6r5\") pod \"e183677e-396a-41f6-aa42-1184c82d520b\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") "
Oct 31 12:53:16.153910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.153751    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-certs\") pod \"e183677e-396a-41f6-aa42-1184c82d520b\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") "
Oct 31 12:53:16.153910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.153809    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-socket\") pod \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") "
Oct 31 12:53:16.153910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.153815    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-certs" pod "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" (UID: "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:16.154215 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.154190    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-socket" pod "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" (UID: "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:16.154215 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.154207    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-credential-socket" pod "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" (UID: "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:16.154406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.154325    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e183677e-396a-41f6-aa42-1184c82d520b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e183677e-396a-41f6-aa42-1184c82d520b-istiod-ca-cert\") pod \"e183677e-396a-41f6-aa42-1184c82d520b\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") "
Oct 31 12:53:16.154406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.154392    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-envoy\") pod \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") "
Oct 31 12:53:16.154630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.154508    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-certs" pod "e183677e-396a-41f6-aa42-1184c82d520b" (UID: "e183677e-396a-41f6-aa42-1184c82d520b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:16.154774 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.154737    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istiod-ca-cert\") pod \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") "
Oct 31 12:53:16.154873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.154812    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e183677e-396a-41f6-aa42-1184c82d520b-istiod-ca-cert" pod "e183677e-396a-41f6-aa42-1184c82d520b" (UID: "e183677e-396a-41f6-aa42-1184c82d520b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:16.154873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.154852    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-socket\") pod \"e183677e-396a-41f6-aa42-1184c82d520b\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") "
Oct 31 12:53:16.155090 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.155017    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-kube-api-access-vqxb4\" (UniqueName: \"kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-kube-api-access-vqxb4\") pod \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") "
Oct 31 12:53:16.155090 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.155055    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-data\") pod \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") "
Oct 31 12:53:16.155090 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.155082    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-credential-socket\") pod \"e183677e-396a-41f6-aa42-1184c82d520b\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") "
Oct 31 12:53:16.155326 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.155120    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-data\") pod \"e183677e-396a-41f6-aa42-1184c82d520b\" (UID: \"e183677e-396a-41f6-aa42-1184c82d520b\") "
Oct 31 12:53:16.155326 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.155153    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-token\" (UniqueName: \"kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-token\") pod \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\" (UID: \"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9\") "
Oct 31 12:53:16.156238 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.156214    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.156343 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.156241    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.156343 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.156252    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.156343 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.156261    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.156343 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.156270    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e183677e-396a-41f6-aa42-1184c82d520b-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.156587 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.156388    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-istio-token" pod "e183677e-396a-41f6-aa42-1184c82d520b" (UID: "e183677e-396a-41f6-aa42-1184c82d520b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:16.156587 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.156485    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-kube-api-access-db6r5" pod "e183677e-396a-41f6-aa42-1184c82d520b" (UID: "e183677e-396a-41f6-aa42-1184c82d520b"). InnerVolumeSpecName "kube-api-access-db6r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:16.156587 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.156544    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-envoy" pod "e183677e-396a-41f6-aa42-1184c82d520b" (UID: "e183677e-396a-41f6-aa42-1184c82d520b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:16.157078 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.156757    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-credential-socket" pod "e183677e-396a-41f6-aa42-1184c82d520b" (UID: "e183677e-396a-41f6-aa42-1184c82d520b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:16.157078 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.156839    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-data" pod "e183677e-396a-41f6-aa42-1184c82d520b" (UID: "e183677e-396a-41f6-aa42-1184c82d520b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:16.157078 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.157037    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-data" pod "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" (UID: "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:16.157078 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.157043    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-socket" pod "e183677e-396a-41f6-aa42-1184c82d520b" (UID: "e183677e-396a-41f6-aa42-1184c82d520b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:16.157798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.157768    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istiod-ca-cert" pod "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" (UID: "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:16.158449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.158424    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-token" pod "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" (UID: "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:16.159068 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.159042    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-kube-api-access-vqxb4" pod "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" (UID: "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9"). InnerVolumeSpecName "kube-api-access-vqxb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:16.159839 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.159819    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-podinfo" pod "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" (UID: "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:16.159995 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.159973    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e183677e-396a-41f6-aa42-1184c82d520b-istio-podinfo" pod "e183677e-396a-41f6-aa42-1184c82d520b" (UID: "e183677e-396a-41f6-aa42-1184c82d520b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:16.160596 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.160573    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-envoy" pod "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" (UID: "15fac6ce-bfc2-44c7-bf67-ddb23982f1e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:16.258021 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.257989    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e183677e-396a-41f6-aa42-1184c82d520b-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.258021 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.258019    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.258021 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.258028    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.258422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.258039    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.258422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.258049    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db6r5\" (UniqueName: \"kubernetes.io/projected/e183677e-396a-41f6-aa42-1184c82d520b-kube-api-access-db6r5\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.258422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.258060    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.258422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.258068    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.258422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.258078    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.258422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.258086    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqxb4\" (UniqueName: \"kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-kube-api-access-vqxb4\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.258422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.258096    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.258422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.258104    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.258422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.258112    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e183677e-396a-41f6-aa42-1184c82d520b-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.258422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.258119    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:16.812333 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.812297    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7" event={"ID":"71f3a982-4cac-4228-b891-dca75a989765","Type":"ContainerStarted","Data":"9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3"}
Oct 31 12:53:16.812858 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.812374    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7" event={"ID":"71f3a982-4cac-4228-b891-dca75a989765","Type":"ContainerStarted","Data":"9999e7dedb9938708ba7c34925d52101b22a0f7b231023a8133ccafb6781d670"}
Oct 31 12:53:16.812858 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.812405    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"
Oct 31 12:53:16.813766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.813745    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"
Oct 31 12:53:16.813894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.813785    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6" event={"ID":"e183677e-396a-41f6-aa42-1184c82d520b","Type":"ContainerDied","Data":"658fbefd5e666645e4f803041f2ce918ea7f52c691b1b8c1469aa1da5a0906a3"}
Oct 31 12:53:16.813894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.813835    5663 scope.go:122] "RemoveContainer" containerID="3950c3ffd9b4906c4612d07c7b7758690ae38a84edf1b979cef879cc2bba0ef2"
Oct 31 12:53:16.815426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.815407    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"
Oct 31 12:53:16.815707 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.815410    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98" event={"ID":"a78f4e51-551c-43f8-a199-01ebd2c11e1a","Type":"ContainerDied","Data":"94ab1f1bdb7fadc65983cf7a2fc59cdca9ecd5072c4abe8b6bd9b39d70d36304"}
Oct 31 12:53:16.817065 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.817041    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5" event={"ID":"15fac6ce-bfc2-44c7-bf67-ddb23982f1e9","Type":"ContainerDied","Data":"f725329c7ad125a51439176a6af9f94ef13420f0a6b59c51d7733d0539646f9a"}
Oct 31 12:53:16.817177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.817047    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"
Oct 31 12:53:16.825733 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.825712    5663 scope.go:122] "RemoveContainer" containerID="74475ac74a756ceaad112d68e691f6cfa356dca69a75f9978308ac3d0fffa7e4"
Oct 31 12:53:16.834828 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.834782    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7" podStartSLOduration=1.834768274 podStartE2EDuration="1.834768274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:16.830237141 +0000 UTC m=+1521.570663368" watchObservedRunningTime="2026-10-31 12:53:16.834768274 +0000 UTC m=+1521.575194537"
Oct 31 12:53:16.839783 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.839764    5663 scope.go:122] "RemoveContainer" containerID="455b83d46a21d1112ed78e7f574b4c9eb36231c0682284939f1b034e2b637a99"
Oct 31 12:53:16.852182 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.852159    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"]
Oct 31 12:53:16.856200 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.856178    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-hn9w6"]
Oct 31 12:53:16.866642 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.866622    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"]
Oct 31 12:53:16.873417 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.873393    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-dxx98"]
Oct 31 12:53:16.885294 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.885271    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"]
Oct 31 12:53:16.893802 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:16.893776    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-cnwl5"]
Oct 31 12:53:17.441466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:17.441434    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" path="/var/lib/kubelet/pods/15fac6ce-bfc2-44c7-bf67-ddb23982f1e9/volumes"
Oct 31 12:53:17.442157 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:17.442131    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1778cc89-0d95-46b0-b633-66a43483dc2d" path="/var/lib/kubelet/pods/1778cc89-0d95-46b0-b633-66a43483dc2d/volumes"
Oct 31 12:53:17.442745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:17.442724    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" path="/var/lib/kubelet/pods/a78f4e51-551c-43f8-a199-01ebd2c11e1a/volumes"
Oct 31 12:53:17.443666 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:17.443632    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e183677e-396a-41f6-aa42-1184c82d520b" path="/var/lib/kubelet/pods/e183677e-396a-41f6-aa42-1184c82d520b/volumes"
Oct 31 12:53:20.868924 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:20.868892    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:21.923972 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:21.923937    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"]
Oct 31 12:53:21.924427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:21.924194    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerName="istio-proxy" containerID="cri-o://e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54" gracePeriod=30
Oct 31 12:53:21.927909 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:21.927883    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"]
Oct 31 12:53:21.928079 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:21.928060    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy" containerID="cri-o://47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e" gracePeriod=30
Oct 31 12:53:22.418817 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.418780    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"]
Oct 31 12:53:22.420108 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.420073    5663 cpu_manager.go:414] "containerMap: removing container" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerName="istio-proxy"
Oct 31 12:53:22.420108 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.420107    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerName="istio-proxy"
Oct 31 12:53:22.420310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.420160    5663 cpu_manager.go:414] "containerMap: removing container" podUID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" containerName="istio-proxy"
Oct 31 12:53:22.420310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.420169    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" containerName="istio-proxy"
Oct 31 12:53:22.420310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.420189    5663 cpu_manager.go:414] "containerMap: removing container" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy"
Oct 31 12:53:22.420310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.420198    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy"
Oct 31 12:53:22.420310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.420244    5663 cpu_manager.go:414] "containerMap: removing container" podUID="1778cc89-0d95-46b0-b633-66a43483dc2d" containerName="istio-proxy"
Oct 31 12:53:22.420310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.420251    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1778cc89-0d95-46b0-b633-66a43483dc2d" containerName="istio-proxy"
Oct 31 12:53:22.420637 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.420456    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78f4e51-551c-43f8-a199-01ebd2c11e1a" containerName="istio-proxy"
Oct 31 12:53:22.420637 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.420479    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="15fac6ce-bfc2-44c7-bf67-ddb23982f1e9" containerName="istio-proxy"
Oct 31 12:53:22.420637 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.420491    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="e183677e-396a-41f6-aa42-1184c82d520b" containerName="istio-proxy"
Oct 31 12:53:22.420637 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.420511    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="1778cc89-0d95-46b0-b633-66a43483dc2d" containerName="istio-proxy"
Oct 31 12:53:22.426724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.426699    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:22.431940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.431909    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"]
Oct 31 12:53:22.433917 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.433894    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:53:22.434648 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.434629    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:53:22.451277 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.450842    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn\" (UID: \"0d9001d5-adb8-4358-90fd-0e90f039a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:22.451732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.451707    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn\" (UID: \"0d9001d5-adb8-4358-90fd-0e90f039a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:22.452094 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.452076    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d84bn\" (UniqueName: \"kubernetes.io/projected/0d9001d5-adb8-4358-90fd-0e90f039a272-kube-api-access-d84bn\") pod \"maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn\" (UID: \"0d9001d5-adb8-4358-90fd-0e90f039a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:22.519729 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:53:22.519674    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:53:22.551345 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.551315    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"
Oct 31 12:53:22.553808 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.553784    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn\" (UID: \"0d9001d5-adb8-4358-90fd-0e90f039a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:22.553956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.553876    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn\" (UID: \"0d9001d5-adb8-4358-90fd-0e90f039a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:22.554020 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.554001    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84bn\" (UniqueName: \"kubernetes.io/projected/0d9001d5-adb8-4358-90fd-0e90f039a272-kube-api-access-d84bn\") pod \"maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn\" (UID: \"0d9001d5-adb8-4358-90fd-0e90f039a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:22.554505 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.554483    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn\" (UID: \"0d9001d5-adb8-4358-90fd-0e90f039a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:22.556227 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.556204    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn\" (UID: \"0d9001d5-adb8-4358-90fd-0e90f039a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:22.583265 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.583230    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d84bn\" (UniqueName: \"kubernetes.io/projected/0d9001d5-adb8-4358-90fd-0e90f039a272-kube-api-access-d84bn\") pod \"maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn\" (UID: \"0d9001d5-adb8-4358-90fd-0e90f039a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:22.750873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:22.750764    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:23.070587 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:53:23.070555    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9001d5_adb8_4358_90fd_0e90f039a272.slice/crio-163224f6c036d10aa55dcf45cf0b9b7b2329d2d7809c2de13b6d3799f3574f78 WatchSource:0}: Error finding container 163224f6c036d10aa55dcf45cf0b9b7b2329d2d7809c2de13b6d3799f3574f78: Status 404 returned error can't find the container with id 163224f6c036d10aa55dcf45cf0b9b7b2329d2d7809c2de13b6d3799f3574f78
Oct 31 12:53:23.071142 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:23.071116    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"]
Oct 31 12:53:23.968940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:23.968907    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn" event={"ID":"0d9001d5-adb8-4358-90fd-0e90f039a272","Type":"ContainerStarted","Data":"b294badace00abaa3f7263a8c79a13ac5e6fb79325aa7364782d0c047c0a6e90"}
Oct 31 12:53:23.969119 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:23.968962    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn" event={"ID":"0d9001d5-adb8-4358-90fd-0e90f039a272","Type":"ContainerStarted","Data":"163224f6c036d10aa55dcf45cf0b9b7b2329d2d7809c2de13b6d3799f3574f78"}
Oct 31 12:53:24.236383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:24.233230    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn" podStartSLOduration=2.233208327 podStartE2EDuration="2.233208327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:23.993249468 +0000 UTC m=+1528.733675685" watchObservedRunningTime="2026-10-31 12:53:24.233208327 +0000 UTC m=+1528.973634545"
Oct 31 12:53:24.980230 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:24.980194    5663 generic.go:356] "Generic (PLEG): container finished" podID="0d9001d5-adb8-4358-90fd-0e90f039a272" containerID="b294badace00abaa3f7263a8c79a13ac5e6fb79325aa7364782d0c047c0a6e90" exitCode=0
Oct 31 12:53:24.980449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:24.980281    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn" event={"ID":"0d9001d5-adb8-4358-90fd-0e90f039a272","Type":"ContainerDied","Data":"b294badace00abaa3f7263a8c79a13ac5e6fb79325aa7364782d0c047c0a6e90"}
Oct 31 12:53:26.439805 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.439780    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:26.684276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.684189    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d9001d5-adb8-4358-90fd-0e90f039a272-kube-api-access-d84bn\" (UniqueName: \"kubernetes.io/projected/0d9001d5-adb8-4358-90fd-0e90f039a272-kube-api-access-d84bn\") pod \"0d9001d5-adb8-4358-90fd-0e90f039a272\" (UID: \"0d9001d5-adb8-4358-90fd-0e90f039a272\") "
Oct 31 12:53:26.684276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.684263    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-xks-ca\") pod \"0d9001d5-adb8-4358-90fd-0e90f039a272\" (UID: \"0d9001d5-adb8-4358-90fd-0e90f039a272\") "
Oct 31 12:53:26.684576 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.684405    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-openshift-ca\") pod \"0d9001d5-adb8-4358-90fd-0e90f039a272\" (UID: \"0d9001d5-adb8-4358-90fd-0e90f039a272\") "
Oct 31 12:53:26.684807 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.684773    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-openshift-ca" pod "0d9001d5-adb8-4358-90fd-0e90f039a272" (UID: "0d9001d5-adb8-4358-90fd-0e90f039a272"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:26.685034 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.685013    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:26.686410 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.686372    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-xks-ca" pod "0d9001d5-adb8-4358-90fd-0e90f039a272" (UID: "0d9001d5-adb8-4358-90fd-0e90f039a272"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:53:26.686506 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.686447    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9001d5-adb8-4358-90fd-0e90f039a272-kube-api-access-d84bn" pod "0d9001d5-adb8-4358-90fd-0e90f039a272" (UID: "0d9001d5-adb8-4358-90fd-0e90f039a272"). InnerVolumeSpecName "kube-api-access-d84bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:26.787033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.786998    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d84bn\" (UniqueName: \"kubernetes.io/projected/0d9001d5-adb8-4358-90fd-0e90f039a272-kube-api-access-d84bn\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:26.787033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.787028    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9001d5-adb8-4358-90fd-0e90f039a272-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:26.996485 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.996457    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"
Oct 31 12:53:26.996609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.996497    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn" event={"ID":"0d9001d5-adb8-4358-90fd-0e90f039a272","Type":"ContainerDied","Data":"163224f6c036d10aa55dcf45cf0b9b7b2329d2d7809c2de13b6d3799f3574f78"}
Oct 31 12:53:26.996609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:26.996549    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="163224f6c036d10aa55dcf45cf0b9b7b2329d2d7809c2de13b6d3799f3574f78"
Oct 31 12:53:27.248730 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.248706    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:53:27.401388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.400123    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:53:27.418181 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.418143    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-certs\") pod \"52e2d1c0-5688-4db4-9630-f40ce094891e\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") "
Oct 31 12:53:27.418427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.418256    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-podinfo\") pod \"52e2d1c0-5688-4db4-9630-f40ce094891e\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") "
Oct 31 12:53:27.418427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.418333    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-data\") pod \"52e2d1c0-5688-4db4-9630-f40ce094891e\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") "
Oct 31 12:53:27.418560 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.418470    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-credential-socket\") pod \"52e2d1c0-5688-4db4-9630-f40ce094891e\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") "
Oct 31 12:53:27.418560 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.418546    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-token\" (UniqueName: \"kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-token\") pod \"52e2d1c0-5688-4db4-9630-f40ce094891e\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") "
Oct 31 12:53:27.418671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.418592    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-kube-api-access-bnpg8\" (UniqueName: \"kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-kube-api-access-bnpg8\") pod \"52e2d1c0-5688-4db4-9630-f40ce094891e\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") "
Oct 31 12:53:27.418731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.418710    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-envoy\") pod \"52e2d1c0-5688-4db4-9630-f40ce094891e\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") "
Oct 31 12:53:27.418786 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.418771    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-socket\") pod \"52e2d1c0-5688-4db4-9630-f40ce094891e\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") "
Oct 31 12:53:27.423694 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.418851    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52e2d1c0-5688-4db4-9630-f40ce094891e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e2d1c0-5688-4db4-9630-f40ce094891e-istiod-ca-cert\") pod \"52e2d1c0-5688-4db4-9630-f40ce094891e\" (UID: \"52e2d1c0-5688-4db4-9630-f40ce094891e\") "
Oct 31 12:53:27.423694 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.420135    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e2d1c0-5688-4db4-9630-f40ce094891e-istiod-ca-cert" pod "52e2d1c0-5688-4db4-9630-f40ce094891e" (UID: "52e2d1c0-5688-4db4-9630-f40ce094891e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:27.423694 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.420464    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-certs" pod "52e2d1c0-5688-4db4-9630-f40ce094891e" (UID: "52e2d1c0-5688-4db4-9630-f40ce094891e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:27.423694 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.421219    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-data" pod "52e2d1c0-5688-4db4-9630-f40ce094891e" (UID: "52e2d1c0-5688-4db4-9630-f40ce094891e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:27.423694 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.422066    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-socket" pod "52e2d1c0-5688-4db4-9630-f40ce094891e" (UID: "52e2d1c0-5688-4db4-9630-f40ce094891e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:27.425564 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.425384    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-credential-socket" pod "52e2d1c0-5688-4db4-9630-f40ce094891e" (UID: "52e2d1c0-5688-4db4-9630-f40ce094891e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:27.429838 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.429780    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-token" pod "52e2d1c0-5688-4db4-9630-f40ce094891e" (UID: "52e2d1c0-5688-4db4-9630-f40ce094891e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:27.430259 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.430237    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-envoy" pod "52e2d1c0-5688-4db4-9630-f40ce094891e" (UID: "52e2d1c0-5688-4db4-9630-f40ce094891e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:27.430658 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.430636    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-podinfo" pod "52e2d1c0-5688-4db4-9630-f40ce094891e" (UID: "52e2d1c0-5688-4db4-9630-f40ce094891e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:27.437272 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.437234    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-kube-api-access-bnpg8" pod "52e2d1c0-5688-4db4-9630-f40ce094891e" (UID: "52e2d1c0-5688-4db4-9630-f40ce094891e"). InnerVolumeSpecName "kube-api-access-bnpg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:27.521454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.521308    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-kube-api-access-kz4kd\" (UniqueName: \"kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-kube-api-access-kz4kd\") pod \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") "
Oct 31 12:53:27.521955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.521486    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-credential-socket\") pod \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") "
Oct 31 12:53:27.521955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.521625    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-envoy\") pod \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") "
Oct 31 12:53:27.521955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.521655    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istiod-ca-cert\") pod \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") "
Oct 31 12:53:27.521955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.521693    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-token\" (UniqueName: \"kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-token\") pod \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") "
Oct 31 12:53:27.521955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.521786    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-certs\") pod \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") "
Oct 31 12:53:27.521955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.521918    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-data\") pod \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") "
Oct 31 12:53:27.521955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.521950    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-socket\") pod \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") "
Oct 31 12:53:27.522294 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.522014    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-podinfo\") pod \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\" (UID: \"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d\") "
Oct 31 12:53:27.522371 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.522307    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-credential-socket" pod "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" (UID: "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:27.522710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.522683    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-certs" pod "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" (UID: "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:27.522906 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.522887    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.522984 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.522906    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-socket" pod "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" (UID: "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:27.522984 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.522914    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnpg8\" (UniqueName: \"kubernetes.io/projected/52e2d1c0-5688-4db4-9630-f40ce094891e-kube-api-access-bnpg8\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.522984 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.522939    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istiod-ca-cert" pod "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" (UID: "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:27.522984 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.522954    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.522984 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.522974    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.523248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.523014    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.523248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.523031    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e2d1c0-5688-4db4-9630-f40ce094891e-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.523248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.523045    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.523248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.523059    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.523248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.523072    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.523248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.523085    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.523248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.523098    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e2d1c0-5688-4db4-9630-f40ce094891e-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.523248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.523137    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-data" pod "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" (UID: "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:27.524128 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.523875    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-kube-api-access-kz4kd" pod "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" (UID: "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d"). InnerVolumeSpecName "kube-api-access-kz4kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:27.524313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.524264    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-envoy" pod "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" (UID: "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:53:27.525097 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.525062    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-token" pod "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" (UID: "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:27.525664 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.525643    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-podinfo" pod "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" (UID: "e57f67c5-9c24-4043-9b27-a2f5b1f18d8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:53:27.625421 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.625387    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.625421 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.625421    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.625663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.625436    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.625663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.625450    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.625663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.625466    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz4kd\" (UniqueName: \"kubernetes.io/projected/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-kube-api-access-kz4kd\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.625663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.625481    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.625663 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.625495    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:27.913380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:27.890568    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"
Oct 31 12:53:28.049402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.044186    5663 generic.go:356] "Generic (PLEG): container finished" podID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerID="e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54" exitCode=0
Oct 31 12:53:28.049402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.047239    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"
Oct 31 12:53:28.049402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.047697    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" event={"ID":"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d","Type":"ContainerDied","Data":"e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54"}
Oct 31 12:53:28.049402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.047758    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5" event={"ID":"e57f67c5-9c24-4043-9b27-a2f5b1f18d8d","Type":"ContainerDied","Data":"fee6d3cecb00f7224baacc44bd58c0fee7c82bfba1ff329bd9c4a28424403a2b"}
Oct 31 12:53:28.049402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.047787    5663 scope.go:122] "RemoveContainer" containerID="e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54"
Oct 31 12:53:28.081009 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.068831    5663 generic.go:356] "Generic (PLEG): container finished" podID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerID="47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e" exitCode=0
Oct 31 12:53:28.081009 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.068928    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" event={"ID":"52e2d1c0-5688-4db4-9630-f40ce094891e","Type":"ContainerDied","Data":"47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e"}
Oct 31 12:53:28.081009 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.068983    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m" event={"ID":"52e2d1c0-5688-4db4-9630-f40ce094891e","Type":"ContainerDied","Data":"2e8632af93a3d54d6781fd94702d1986ecd1c3c8d8bea7aab72736c7ca52d6bc"}
Oct 31 12:53:28.081009 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.069070    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"
Oct 31 12:53:28.092420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.092312    5663 scope.go:122] "RemoveContainer" containerID="e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54"
Oct 31 12:53:28.093723 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:53:28.092776    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54\": container with ID starting with e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54 not found: ID does not exist" containerID="e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54"
Oct 31 12:53:28.093723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.092814    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54"} err="failed to get container status \"e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54\": rpc error: code = NotFound desc = could not find container \"e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54\": container with ID starting with e68f0e5627287237767a0867ef82c6221a8bb3330480b3e59cafaa88f0f56c54 not found: ID does not exist"
Oct 31 12:53:28.093723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.092855    5663 scope.go:122] "RemoveContainer" containerID="47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e"
Oct 31 12:53:28.165970 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.162319    5663 scope.go:122] "RemoveContainer" containerID="47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e"
Oct 31 12:53:28.165970 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:53:28.162843    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e\": container with ID starting with 47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e not found: ID does not exist" containerID="47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e"
Oct 31 12:53:28.165970 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.162900    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e"} err="failed to get container status \"47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e\": rpc error: code = NotFound desc = could not find container \"47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e\": container with ID starting with 47fc0b1fa19eba14d2cb22d992aeac1e492c7ce4618c0fb3dca9a9b8fa51a05e not found: ID does not exist"
Oct 31 12:53:28.194656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.188776    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"]
Oct 31 12:53:28.194656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.189903    5663 cpu_manager.go:414] "containerMap: removing container" podUID="0d9001d5-adb8-4358-90fd-0e90f039a272" containerName="revoke-keys"
Oct 31 12:53:28.194656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.189930    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d9001d5-adb8-4358-90fd-0e90f039a272" containerName="revoke-keys"
Oct 31 12:53:28.194656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.189997    5663 cpu_manager.go:414] "containerMap: removing container" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerName="istio-proxy"
Oct 31 12:53:28.194656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.190006    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerName="istio-proxy"
Oct 31 12:53:28.194656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.190041    5663 cpu_manager.go:414] "containerMap: removing container" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy"
Oct 31 12:53:28.194656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.190049    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy"
Oct 31 12:53:28.194656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.190278    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" containerName="istio-proxy"
Oct 31 12:53:28.194656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.190301    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" containerName="istio-proxy"
Oct 31 12:53:28.194656 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.190315    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d9001d5-adb8-4358-90fd-0e90f039a272" containerName="revoke-keys"
Oct 31 12:53:28.204662 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.197522    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.208464 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.205369    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d176a5-serving-cert\""
Oct 31 12:53:28.229762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.229734    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"]
Oct 31 12:53:28.235220 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.235197    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"]
Oct 31 12:53:28.242007 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.241982    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-kqn9m"]
Oct 31 12:53:28.245589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.245569    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"]
Oct 31 12:53:28.258748 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.258722    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4e8eb9-openshift-default-768d574d85-w8fz5"]
Oct 31 12:53:28.305908 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.305877    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52c4w\" (UniqueName: \"kubernetes.io/projected/2ee276b8-0de2-4586-9a97-40cc65048ea5-kube-api-access-52c4w\") pod \"maas-api-e2e-inf-a-d176a5-7b698689b-vknb7\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.306065 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.305926    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-maas-api-tls\") pod \"maas-api-e2e-inf-a-d176a5-7b698689b-vknb7\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.306267 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.306244    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-metrics-certs\") pod \"maas-api-e2e-inf-a-d176a5-7b698689b-vknb7\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.306310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.306294    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee276b8-0de2-4586-9a97-40cc65048ea5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d176a5-7b698689b-vknb7\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.429515 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.418097    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-metrics-certs\") pod \"maas-api-e2e-inf-a-d176a5-7b698689b-vknb7\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.429515 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.418184    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee276b8-0de2-4586-9a97-40cc65048ea5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d176a5-7b698689b-vknb7\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.429515 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.418398    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52c4w\" (UniqueName: \"kubernetes.io/projected/2ee276b8-0de2-4586-9a97-40cc65048ea5-kube-api-access-52c4w\") pod \"maas-api-e2e-inf-a-d176a5-7b698689b-vknb7\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.429515 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.418443    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-maas-api-tls\") pod \"maas-api-e2e-inf-a-d176a5-7b698689b-vknb7\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.431754 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.431717    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee276b8-0de2-4586-9a97-40cc65048ea5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d176a5-7b698689b-vknb7\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.440403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.440370    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-metrics-certs\") pod \"maas-api-e2e-inf-a-d176a5-7b698689b-vknb7\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.444637 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.444564    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-maas-api-tls\") pod \"maas-api-e2e-inf-a-d176a5-7b698689b-vknb7\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.452375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.451335    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52c4w\" (UniqueName: \"kubernetes.io/projected/2ee276b8-0de2-4586-9a97-40cc65048ea5-kube-api-access-52c4w\") pod \"maas-api-e2e-inf-a-d176a5-7b698689b-vknb7\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:28.536997 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:28.536964    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:29.075384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:29.065945    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"]
Oct 31 12:53:29.442091 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:29.442016    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e2d1c0-5688-4db4-9630-f40ce094891e" path="/var/lib/kubelet/pods/52e2d1c0-5688-4db4-9630-f40ce094891e/volumes"
Oct 31 12:53:29.442626 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:29.442609    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57f67c5-9c24-4043-9b27-a2f5b1f18d8d" path="/var/lib/kubelet/pods/e57f67c5-9c24-4043-9b27-a2f5b1f18d8d/volumes"
Oct 31 12:53:30.183780 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:30.183739    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7" event={"ID":"2ee276b8-0de2-4586-9a97-40cc65048ea5","Type":"ContainerStarted","Data":"3598f0f69399e39594b68420765568620debb5846aa6da0e04c5376e14943ffa"}
Oct 31 12:53:30.184266 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:30.183814    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:30.184266 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:30.183838    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7" event={"ID":"2ee276b8-0de2-4586-9a97-40cc65048ea5","Type":"ContainerStarted","Data":"a3b186cc003ce33e44668cb3b85d2fe7878b46697e1b6d7854395403c87dd41a"}
Oct 31 12:53:30.207818 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:30.207768    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7" podStartSLOduration=1.709708883 podStartE2EDuration="2.207754867s" totalImagesPullingTime="498.045984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:30.204893671 +0000 UTC m=+1534.945319900" watchObservedRunningTime="2026-10-31 12:53:30.207754867 +0000 UTC m=+1534.948181083"
Oct 31 12:53:34.628534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:34.622774    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"]
Oct 31 12:53:34.628534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:34.623167    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x" podUID="89b18aea-9ac2-44bb-8c98-6de55fc5e1eb" containerName="maas-api" containerID="cri-o://aaa746df2bcca24f66c38b6454ddbd8bc9805ffee7e58fa138f9929514fc6a84" gracePeriod=30
Oct 31 12:53:35.290873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.290836    5663 generic.go:356] "Generic (PLEG): container finished" podID="89b18aea-9ac2-44bb-8c98-6de55fc5e1eb" containerID="aaa746df2bcca24f66c38b6454ddbd8bc9805ffee7e58fa138f9929514fc6a84" exitCode=0
Oct 31 12:53:35.306886 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.306840    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x" event={"ID":"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb","Type":"ContainerDied","Data":"aaa746df2bcca24f66c38b6454ddbd8bc9805ffee7e58fa138f9929514fc6a84"}
Oct 31 12:53:35.311440 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.311403    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"]
Oct 31 12:53:35.375838 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.375792    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a0ea4-7ccwn"]
Oct 31 12:53:35.461487 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.461435    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d9001d5-adb8-4358-90fd-0e90f039a272" path="/var/lib/kubelet/pods/0d9001d5-adb8-4358-90fd-0e90f039a272/volumes"
Oct 31 12:53:35.667792 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.667768    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:35.719514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.719481    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-kube-api-access-6p5vh\" (UniqueName: \"kubernetes.io/projected/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-kube-api-access-6p5vh\") pod \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") "
Oct 31 12:53:35.719716 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.719696    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-maas-api-tls\") pod \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") "
Oct 31 12:53:35.719875 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.719858    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-metrics-certs\") pod \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") "
Oct 31 12:53:35.719944 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.719897    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-openshift-service-ca\") pod \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\" (UID: \"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb\") "
Oct 31 12:53:35.720373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.720326    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-openshift-service-ca" pod "89b18aea-9ac2-44bb-8c98-6de55fc5e1eb" (UID: "89b18aea-9ac2-44bb-8c98-6de55fc5e1eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:35.720703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.720675    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:35.721824 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.721798    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-kube-api-access-6p5vh" pod "89b18aea-9ac2-44bb-8c98-6de55fc5e1eb" (UID: "89b18aea-9ac2-44bb-8c98-6de55fc5e1eb"). InnerVolumeSpecName "kube-api-access-6p5vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:35.721928 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.721894    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-maas-api-tls" pod "89b18aea-9ac2-44bb-8c98-6de55fc5e1eb" (UID: "89b18aea-9ac2-44bb-8c98-6de55fc5e1eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:53:35.722520 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.722491    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-metrics-certs" pod "89b18aea-9ac2-44bb-8c98-6de55fc5e1eb" (UID: "89b18aea-9ac2-44bb-8c98-6de55fc5e1eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:53:35.823425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.823211    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:35.823425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.823261    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p5vh\" (UniqueName: \"kubernetes.io/projected/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-kube-api-access-6p5vh\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:35.823425 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:35.823278    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:36.083629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:36.083553    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"]
Oct 31 12:53:36.197784 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:36.197755    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:53:36.316427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:36.316387    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x" event={"ID":"89b18aea-9ac2-44bb-8c98-6de55fc5e1eb","Type":"ContainerDied","Data":"8381a61078c320e0134e6437307a7809377d9ebb73f784d00d556bb849b20da4"}
Oct 31 12:53:36.316602 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:36.316433    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"
Oct 31 12:53:36.316602 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:36.316464    5663 scope.go:122] "RemoveContainer" containerID="aaa746df2bcca24f66c38b6454ddbd8bc9805ffee7e58fa138f9929514fc6a84"
Oct 31 12:53:36.359336 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:36.359264    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"]
Oct 31 12:53:36.371671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:36.371646    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a0ea4-6cb88f89d8-j848x"]
Oct 31 12:53:37.522458 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:37.517853    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b18aea-9ac2-44bb-8c98-6de55fc5e1eb" path="/var/lib/kubelet/pods/89b18aea-9ac2-44bb-8c98-6de55fc5e1eb/volumes"
Oct 31 12:53:41.084447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:41.084391    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q" podUID="eba68b32-a972-45c1-aabe-f769ad8113a9" containerName="payload-pre-processing" containerID="cri-o://c6fd56c85e98da1c1b89d98ffc4d26df21702bf17dd3bff3969e398a4085b0a5" gracePeriod=25
Oct 31 12:53:41.355629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:41.355591    5663 generic.go:356] "Generic (PLEG): container finished" podID="eba68b32-a972-45c1-aabe-f769ad8113a9" containerID="c6fd56c85e98da1c1b89d98ffc4d26df21702bf17dd3bff3969e398a4085b0a5" exitCode=0
Oct 31 12:53:41.355780 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:41.355626    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q" event={"ID":"eba68b32-a972-45c1-aabe-f769ad8113a9","Type":"ContainerDied","Data":"c6fd56c85e98da1c1b89d98ffc4d26df21702bf17dd3bff3969e398a4085b0a5"}
Oct 31 12:53:41.355780 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:41.355741    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q" event={"ID":"eba68b32-a972-45c1-aabe-f769ad8113a9","Type":"ContainerDied","Data":"8f66deaf4add3dbf594525614c5d76bc39fc23366ead11941f2e1bc25f2d62c8"}
Oct 31 12:53:41.355780 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:41.355770    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f66deaf4add3dbf594525614c5d76bc39fc23366ead11941f2e1bc25f2d62c8"
Oct 31 12:53:41.371363 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:41.371330    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"
Oct 31 12:53:41.465082 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:41.465049    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba68b32-a972-45c1-aabe-f769ad8113a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba68b32-a972-45c1-aabe-f769ad8113a9-plugins-config-volume\") pod \"eba68b32-a972-45c1-aabe-f769ad8113a9\" (UID: \"eba68b32-a972-45c1-aabe-f769ad8113a9\") "
Oct 31 12:53:41.465283 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:41.465148    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba68b32-a972-45c1-aabe-f769ad8113a9-kube-api-access-c5gnp\" (UniqueName: \"kubernetes.io/projected/eba68b32-a972-45c1-aabe-f769ad8113a9-kube-api-access-c5gnp\") pod \"eba68b32-a972-45c1-aabe-f769ad8113a9\" (UID: \"eba68b32-a972-45c1-aabe-f769ad8113a9\") "
Oct 31 12:53:41.465506 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:41.465482    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba68b32-a972-45c1-aabe-f769ad8113a9-plugins-config-volume" pod "eba68b32-a972-45c1-aabe-f769ad8113a9" (UID: "eba68b32-a972-45c1-aabe-f769ad8113a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:41.465992 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:41.465964    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba68b32-a972-45c1-aabe-f769ad8113a9-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:41.467835 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:41.467805    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba68b32-a972-45c1-aabe-f769ad8113a9-kube-api-access-c5gnp" pod "eba68b32-a972-45c1-aabe-f769ad8113a9" (UID: "eba68b32-a972-45c1-aabe-f769ad8113a9"). InnerVolumeSpecName "kube-api-access-c5gnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:41.577418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:41.570095    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5gnp\" (UniqueName: \"kubernetes.io/projected/eba68b32-a972-45c1-aabe-f769ad8113a9-kube-api-access-c5gnp\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:42.363210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:42.363183    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"
Oct 31 12:53:42.394478 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:42.394386    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"]
Oct 31 12:53:42.434806 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:42.434771    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1a7d0e-6987dfc8b7-h9v6q"]
Oct 31 12:53:43.022864 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:43.022830    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"]
Oct 31 12:53:43.023139 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:43.023090    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq" podUID="71690981-0f09-4b44-a93f-ce307e7ff025" containerName="maas-api" containerID="cri-o://24ebd9a24d794ceef879837b7fd31913e5770c89160a4a8c86ddd33bde45a322" gracePeriod=30
Oct 31 12:53:43.862444 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:43.861575    5663 generic.go:356] "Generic (PLEG): container finished" podID="71690981-0f09-4b44-a93f-ce307e7ff025" containerID="24ebd9a24d794ceef879837b7fd31913e5770c89160a4a8c86ddd33bde45a322" exitCode=0
Oct 31 12:53:43.916638 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:43.914944    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba68b32-a972-45c1-aabe-f769ad8113a9" path="/var/lib/kubelet/pods/eba68b32-a972-45c1-aabe-f769ad8113a9/volumes"
Oct 31 12:53:43.916638 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:43.915470    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq" event={"ID":"71690981-0f09-4b44-a93f-ce307e7ff025","Type":"ContainerDied","Data":"24ebd9a24d794ceef879837b7fd31913e5770c89160a4a8c86ddd33bde45a322"}
Oct 31 12:53:44.297123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.297093    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:44.395293 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.395254    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71690981-0f09-4b44-a93f-ce307e7ff025-kube-api-access-gbcpb\" (UniqueName: \"kubernetes.io/projected/71690981-0f09-4b44-a93f-ce307e7ff025-kube-api-access-gbcpb\") pod \"71690981-0f09-4b44-a93f-ce307e7ff025\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") "
Oct 31 12:53:44.395508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.395302    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-metrics-certs\") pod \"71690981-0f09-4b44-a93f-ce307e7ff025\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") "
Oct 31 12:53:44.395572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.395510    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-maas-api-tls\") pod \"71690981-0f09-4b44-a93f-ce307e7ff025\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") "
Oct 31 12:53:44.395758 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.395741    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71690981-0f09-4b44-a93f-ce307e7ff025-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71690981-0f09-4b44-a93f-ce307e7ff025-openshift-service-ca\") pod \"71690981-0f09-4b44-a93f-ce307e7ff025\" (UID: \"71690981-0f09-4b44-a93f-ce307e7ff025\") "
Oct 31 12:53:44.396187 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.396157    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71690981-0f09-4b44-a93f-ce307e7ff025-openshift-service-ca" pod "71690981-0f09-4b44-a93f-ce307e7ff025" (UID: "71690981-0f09-4b44-a93f-ce307e7ff025"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:53:44.396588 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.396566    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71690981-0f09-4b44-a93f-ce307e7ff025-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:44.397543 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.397517    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71690981-0f09-4b44-a93f-ce307e7ff025-kube-api-access-gbcpb" pod "71690981-0f09-4b44-a93f-ce307e7ff025" (UID: "71690981-0f09-4b44-a93f-ce307e7ff025"). InnerVolumeSpecName "kube-api-access-gbcpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:53:44.397640 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.397547    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-metrics-certs" pod "71690981-0f09-4b44-a93f-ce307e7ff025" (UID: "71690981-0f09-4b44-a93f-ce307e7ff025"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:53:44.397640 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.397569    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-maas-api-tls" pod "71690981-0f09-4b44-a93f-ce307e7ff025" (UID: "71690981-0f09-4b44-a93f-ce307e7ff025"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:53:44.498609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.498567    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbcpb\" (UniqueName: \"kubernetes.io/projected/71690981-0f09-4b44-a93f-ce307e7ff025-kube-api-access-gbcpb\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:44.498609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.498602    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:44.498609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.498615    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71690981-0f09-4b44-a93f-ce307e7ff025-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:53:44.965375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.949862    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq" event={"ID":"71690981-0f09-4b44-a93f-ce307e7ff025","Type":"ContainerDied","Data":"d2a482db188b65bbf81bb217546417d090746b746721e4cc5adf2b23bda6af26"}
Oct 31 12:53:44.965375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.949933    5663 scope.go:122] "RemoveContainer" containerID="24ebd9a24d794ceef879837b7fd31913e5770c89160a4a8c86ddd33bde45a322"
Oct 31 12:53:44.965375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:44.950065    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"
Oct 31 12:53:45.219859 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:45.219744    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"]
Oct 31 12:53:45.220082 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:45.220069    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e98df8-67dbd7d77b-wsltq"]
Oct 31 12:53:45.442444 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:45.442405    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71690981-0f09-4b44-a93f-ce307e7ff025" path="/var/lib/kubelet/pods/71690981-0f09-4b44-a93f-ce307e7ff025/volumes"
Oct 31 12:53:49.229925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:49.229877    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"]
Oct 31 12:53:49.235893 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:49.235863    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"]
Oct 31 12:53:52.595778 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:53:52.595739    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:53:54.764654 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:54.760892    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"]
Oct 31 12:53:54.774267 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:54.774207    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps" podUID="a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" containerName="istio-proxy" containerID="cri-o://11c826778ba2532b5801dff6a58fbf665ff1905994dae760878abdcd00c7faeb" gracePeriod=30
Oct 31 12:53:55.038632 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:55.037908    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"]
Oct 31 12:53:56.568906 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.568855    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"]
Oct 31 12:53:56.569939 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.569913    5663 cpu_manager.go:414] "containerMap: removing container" podUID="eba68b32-a972-45c1-aabe-f769ad8113a9" containerName="payload-pre-processing"
Oct 31 12:53:56.569939 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.569941    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba68b32-a972-45c1-aabe-f769ad8113a9" containerName="payload-pre-processing"
Oct 31 12:53:56.570133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.569958    5663 cpu_manager.go:414] "containerMap: removing container" podUID="71690981-0f09-4b44-a93f-ce307e7ff025" containerName="maas-api"
Oct 31 12:53:56.570133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.569967    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71690981-0f09-4b44-a93f-ce307e7ff025" containerName="maas-api"
Oct 31 12:53:56.570133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.570034    5663 cpu_manager.go:414] "containerMap: removing container" podUID="89b18aea-9ac2-44bb-8c98-6de55fc5e1eb" containerName="maas-api"
Oct 31 12:53:56.570133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.570043    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b18aea-9ac2-44bb-8c98-6de55fc5e1eb" containerName="maas-api"
Oct 31 12:53:56.570403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.570234    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba68b32-a972-45c1-aabe-f769ad8113a9" containerName="payload-pre-processing"
Oct 31 12:53:56.570403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.570246    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="71690981-0f09-4b44-a93f-ce307e7ff025" containerName="maas-api"
Oct 31 12:53:56.570403 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.570267    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b18aea-9ac2-44bb-8c98-6de55fc5e1eb" containerName="maas-api"
Oct 31 12:53:56.576467 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.576444    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"
Oct 31 12:53:56.588487 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.587254    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"]
Oct 31 12:53:56.602760 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.602729    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2c393f\""
Oct 31 12:53:56.602959 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.602851    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2c393f-dockercfg-9xxr2\""
Oct 31 12:53:56.625105 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.625076    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"]
Oct 31 12:53:56.630077 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.630054    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"
Oct 31 12:53:56.660187 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.660079    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"]
Oct 31 12:53:56.672070 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.672031    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c58e1e6-909c-47d7-98f1-508b00de8fd5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts\" (UID: \"9c58e1e6-909c-47d7-98f1-508b00de8fd5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"
Oct 31 12:53:56.672281 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.672258    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4hzz\" (UniqueName: \"kubernetes.io/projected/9c58e1e6-909c-47d7-98f1-508b00de8fd5-kube-api-access-q4hzz\") pod \"payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts\" (UID: \"9c58e1e6-909c-47d7-98f1-508b00de8fd5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"
Oct 31 12:53:56.672498 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.672474    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mdlz\" (UniqueName: \"kubernetes.io/projected/aa472831-ecc0-4dbb-9bcf-4aec35346832-kube-api-access-9mdlz\") pod \"payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt\" (UID: \"aa472831-ecc0-4dbb-9bcf-4aec35346832\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"
Oct 31 12:53:56.672567 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.672531    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa472831-ecc0-4dbb-9bcf-4aec35346832-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt\" (UID: \"aa472831-ecc0-4dbb-9bcf-4aec35346832\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"
Oct 31 12:53:56.793378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.783508    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mdlz\" (UniqueName: \"kubernetes.io/projected/aa472831-ecc0-4dbb-9bcf-4aec35346832-kube-api-access-9mdlz\") pod \"payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt\" (UID: \"aa472831-ecc0-4dbb-9bcf-4aec35346832\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"
Oct 31 12:53:56.793378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.783587    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa472831-ecc0-4dbb-9bcf-4aec35346832-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt\" (UID: \"aa472831-ecc0-4dbb-9bcf-4aec35346832\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"
Oct 31 12:53:56.793378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.783763    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c58e1e6-909c-47d7-98f1-508b00de8fd5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts\" (UID: \"9c58e1e6-909c-47d7-98f1-508b00de8fd5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"
Oct 31 12:53:56.793378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.783917    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4hzz\" (UniqueName: \"kubernetes.io/projected/9c58e1e6-909c-47d7-98f1-508b00de8fd5-kube-api-access-q4hzz\") pod \"payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts\" (UID: \"9c58e1e6-909c-47d7-98f1-508b00de8fd5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"
Oct 31 12:53:56.793378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.785286    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa472831-ecc0-4dbb-9bcf-4aec35346832-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt\" (UID: \"aa472831-ecc0-4dbb-9bcf-4aec35346832\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"
Oct 31 12:53:56.818376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.812566    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c58e1e6-909c-47d7-98f1-508b00de8fd5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts\" (UID: \"9c58e1e6-909c-47d7-98f1-508b00de8fd5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"
Oct 31 12:53:56.887430 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.866717    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4hzz\" (UniqueName: \"kubernetes.io/projected/9c58e1e6-909c-47d7-98f1-508b00de8fd5-kube-api-access-q4hzz\") pod \"payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts\" (UID: \"9c58e1e6-909c-47d7-98f1-508b00de8fd5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"
Oct 31 12:53:56.924379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.921250    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mdlz\" (UniqueName: \"kubernetes.io/projected/aa472831-ecc0-4dbb-9bcf-4aec35346832-kube-api-access-9mdlz\") pod \"payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt\" (UID: \"aa472831-ecc0-4dbb-9bcf-4aec35346832\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"
Oct 31 12:53:56.954407 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:56.953662    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"
Oct 31 12:53:57.561942 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:57.557774    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"
Oct 31 12:53:57.726637 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:57.723774    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"]
Oct 31 12:53:57.759657 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:57.759062    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 12:53:57.762992 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:57.762961    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"
Oct 31 12:53:57.773012 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:57.772425    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fd988c\""
Oct 31 12:53:57.838391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:57.837509    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfbvt\" (UniqueName: \"kubernetes.io/projected/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-kube-api-access-cfbvt\") pod \"payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c\" (UID: \"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"
Oct 31 12:53:57.838391 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:57.837711    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c\" (UID: \"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"
Oct 31 12:53:57.852375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:57.852021    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"]
Oct 31 12:53:58.024739 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:57.954756    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfbvt\" (UniqueName: \"kubernetes.io/projected/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-kube-api-access-cfbvt\") pod \"payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c\" (UID: \"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"
Oct 31 12:53:58.024739 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:57.954882    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c\" (UID: \"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"
Oct 31 12:53:58.024739 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:57.955991    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c\" (UID: \"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"
Oct 31 12:53:58.230549 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.228273    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"]
Oct 31 12:53:58.256319 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.255228    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"
Oct 31 12:53:58.265377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.262185    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fd988c-dockercfg-9fkr8\""
Oct 31 12:53:58.312063 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.312024    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfbvt\" (UniqueName: \"kubernetes.io/projected/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-kube-api-access-cfbvt\") pod \"payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c\" (UID: \"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"
Oct 31 12:53:58.333607 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.331129    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww7w6\" (UniqueName: \"kubernetes.io/projected/599d803e-9606-46d0-a5bd-d7b5442e1dc2-kube-api-access-ww7w6\") pod \"payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh\" (UID: \"599d803e-9606-46d0-a5bd-d7b5442e1dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"
Oct 31 12:53:58.333607 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.331236    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599d803e-9606-46d0-a5bd-d7b5442e1dc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh\" (UID: \"599d803e-9606-46d0-a5bd-d7b5442e1dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"
Oct 31 12:53:58.354159 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.354123    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"]
Oct 31 12:53:58.359246 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.359208    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"]
Oct 31 12:53:58.409922 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.409890    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts" event={"ID":"9c58e1e6-909c-47d7-98f1-508b00de8fd5","Type":"ContainerStarted","Data":"ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac"}
Oct 31 12:53:58.410047 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.409957    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts" event={"ID":"9c58e1e6-909c-47d7-98f1-508b00de8fd5","Type":"ContainerStarted","Data":"e43f3e10513e04c2e0a3016842c55f5486a574e99ae346e810987ce3c43036f4"}
Oct 31 12:53:58.410198 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.410176    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"]
Oct 31 12:53:58.410279 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.410267    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"
Oct 31 12:53:58.412002 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:53:58.411723    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa472831_ecc0_4dbb_9bcf_4aec35346832.slice/crio-02c1ef274c08a014c8aeeb04fb23103012a31baa0070a48199d7dade3ee20e13 WatchSource:0}: Error finding container 02c1ef274c08a014c8aeeb04fb23103012a31baa0070a48199d7dade3ee20e13: Status 404 returned error can't find the container with id 02c1ef274c08a014c8aeeb04fb23103012a31baa0070a48199d7dade3ee20e13
Oct 31 12:53:58.433534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.433502    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7w6\" (UniqueName: \"kubernetes.io/projected/599d803e-9606-46d0-a5bd-d7b5442e1dc2-kube-api-access-ww7w6\") pod \"payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh\" (UID: \"599d803e-9606-46d0-a5bd-d7b5442e1dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"
Oct 31 12:53:58.433648 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.433617    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599d803e-9606-46d0-a5bd-d7b5442e1dc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh\" (UID: \"599d803e-9606-46d0-a5bd-d7b5442e1dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"
Oct 31 12:53:58.434594 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.434566    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599d803e-9606-46d0-a5bd-d7b5442e1dc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh\" (UID: \"599d803e-9606-46d0-a5bd-d7b5442e1dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"
Oct 31 12:53:58.434773 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.434722    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts" podStartSLOduration=2.434707998 podStartE2EDuration="2.434707998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:58.42851455 +0000 UTC m=+1563.168940767" watchObservedRunningTime="2026-10-31 12:53:58.434707998 +0000 UTC m=+1563.175134216"
Oct 31 12:53:58.442712 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.442654    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"
Oct 31 12:53:58.447827 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.447792    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww7w6\" (UniqueName: \"kubernetes.io/projected/599d803e-9606-46d0-a5bd-d7b5442e1dc2-kube-api-access-ww7w6\") pod \"payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh\" (UID: \"599d803e-9606-46d0-a5bd-d7b5442e1dc2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"
Oct 31 12:53:58.612410 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.609927    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"
Oct 31 12:53:58.757092 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:58.757064    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"]
Oct 31 12:53:58.761661 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:53:58.760234    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01469039_f55a_4ea0_ac8d_e5c8b9a3daaa.slice/crio-637c7e9a7108af67b8dd0737999f4fe1811e5cc7e41af83cfd71614d3cd343ae WatchSource:0}: Error finding container 637c7e9a7108af67b8dd0737999f4fe1811e5cc7e41af83cfd71614d3cd343ae: Status 404 returned error can't find the container with id 637c7e9a7108af67b8dd0737999f4fe1811e5cc7e41af83cfd71614d3cd343ae
Oct 31 12:53:59.050108 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.049847    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"]
Oct 31 12:53:59.423228 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.423135    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt" event={"ID":"aa472831-ecc0-4dbb-9bcf-4aec35346832","Type":"ContainerStarted","Data":"13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180"}
Oct 31 12:53:59.423228 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.423206    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt" event={"ID":"aa472831-ecc0-4dbb-9bcf-4aec35346832","Type":"ContainerStarted","Data":"02c1ef274c08a014c8aeeb04fb23103012a31baa0070a48199d7dade3ee20e13"}
Oct 31 12:53:59.423508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.423236    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"
Oct 31 12:53:59.425184 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.425147    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh" event={"ID":"599d803e-9606-46d0-a5bd-d7b5442e1dc2","Type":"ContainerStarted","Data":"7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c"}
Oct 31 12:53:59.425313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.425208    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh" event={"ID":"599d803e-9606-46d0-a5bd-d7b5442e1dc2","Type":"ContainerStarted","Data":"fb199aa0db80543799b391242d9c424492d4557c26083b6234bc6d881eb2e650"}
Oct 31 12:53:59.425313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.425240    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"
Oct 31 12:53:59.427013 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.426992    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c" event={"ID":"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa","Type":"ContainerStarted","Data":"023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98"}
Oct 31 12:53:59.427134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.427031    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c" event={"ID":"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa","Type":"ContainerStarted","Data":"637c7e9a7108af67b8dd0737999f4fe1811e5cc7e41af83cfd71614d3cd343ae"}
Oct 31 12:53:59.427134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.427086    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"
Oct 31 12:53:59.447535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.447475    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt" podStartSLOduration=3.447455945 podStartE2EDuration="3.447455945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:59.446596006 +0000 UTC m=+1564.187022223" watchObservedRunningTime="2026-10-31 12:53:59.447455945 +0000 UTC m=+1564.187882161"
Oct 31 12:53:59.475024 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.474962    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c" podStartSLOduration=2.474942752 podStartE2EDuration="2.474942752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:59.4705411 +0000 UTC m=+1564.210967316" watchObservedRunningTime="2026-10-31 12:53:59.474942752 +0000 UTC m=+1564.215368970"
Oct 31 12:53:59.498688 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:53:59.498633    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh" podStartSLOduration=2.498617851 podStartE2EDuration="2.498617851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:53:59.496706563 +0000 UTC m=+1564.237132781" watchObservedRunningTime="2026-10-31 12:53:59.498617851 +0000 UTC m=+1564.239044067"
Oct 31 12:54:00.038928 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.038881    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx" podUID="97a2a833-a8a0-499d-8784-310253f03010" containerName="payload-pre-processing" containerID="cri-o://77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011" gracePeriod=25
Oct 31 12:54:00.445163 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.443068    5663 generic.go:356] "Generic (PLEG): container finished" podID="a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" containerID="11c826778ba2532b5801dff6a58fbf665ff1905994dae760878abdcd00c7faeb" exitCode=0
Oct 31 12:54:00.445163 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.444335    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps" event={"ID":"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1","Type":"ContainerDied","Data":"11c826778ba2532b5801dff6a58fbf665ff1905994dae760878abdcd00c7faeb"}
Oct 31 12:54:00.590374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.589689    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:54:00.689775 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.689672    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-credential-socket\") pod \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") "
Oct 31 12:54:00.689948 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.689902    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-token\" (UniqueName: \"kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-token\") pod \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") "
Oct 31 12:54:00.690037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.690007    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-credential-socket" pod "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" (UID: "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:00.690105 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.690036    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-podinfo\") pod \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") "
Oct 31 12:54:00.690500 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.690470    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istiod-ca-cert\") pod \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") "
Oct 31 12:54:00.690594 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.690543    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-data\") pod \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") "
Oct 31 12:54:00.690646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.690620    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-certs\") pod \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") "
Oct 31 12:54:00.690782 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.690749    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-socket\") pod \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") "
Oct 31 12:54:00.690888 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.690800    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-kube-api-access-hhwwz\" (UniqueName: \"kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-kube-api-access-hhwwz\") pod \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") "
Oct 31 12:54:00.690888 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.690851    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-envoy\") pod \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\" (UID: \"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1\") "
Oct 31 12:54:00.691083 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.691067    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-certs" pod "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" (UID: "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:00.691212 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.691126    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-socket" pod "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" (UID: "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:00.691418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.691344    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-data" pod "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" (UID: "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:00.691716 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.691691    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istiod-ca-cert" pod "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" (UID: "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:00.692065 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.692025    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:00.692065 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.692055    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:00.692234 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.692072    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:00.692234 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.692087    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:00.692234 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.692101    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:00.694436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.694402    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-envoy" pod "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" (UID: "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:00.694601 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.694575    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-kube-api-access-hhwwz" pod "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" (UID: "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1"). InnerVolumeSpecName "kube-api-access-hhwwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:00.695136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.695101    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-token" pod "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" (UID: "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:00.696591 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.696566    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-podinfo" pod "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" (UID: "a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:54:00.793300 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.793262    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhwwz\" (UniqueName: \"kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-kube-api-access-hhwwz\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:00.793300 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.793297    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:00.793300 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.793307    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:00.793624 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:00.793316    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:01.064095 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:01.064060    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"]
Oct 31 12:54:01.105335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:01.105304    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"]
Oct 31 12:54:01.127818 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:01.127786    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"]
Oct 31 12:54:01.458547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:01.458519    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"
Oct 31 12:54:01.458701 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:01.458523    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps" event={"ID":"a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1","Type":"ContainerDied","Data":"5ea5f0e5559ab9ece4a4e640688267ccc3ec2807d2b49e36d58c59a95299e501"}
Oct 31 12:54:01.458701 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:01.458656    5663 scope.go:122] "RemoveContainer" containerID="11c826778ba2532b5801dff6a58fbf665ff1905994dae760878abdcd00c7faeb"
Oct 31 12:54:01.492482 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:01.492440    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"]
Oct 31 12:54:01.499707 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:01.499677    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-lx9ps"]
Oct 31 12:54:03.445630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.445594    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" path="/var/lib/kubelet/pods/a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1/volumes"
Oct 31 12:54:03.662955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.662922    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"]
Oct 31 12:54:03.663669 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.663653    5663 cpu_manager.go:414] "containerMap: removing container" podUID="a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" containerName="istio-proxy"
Oct 31 12:54:03.663730 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.663671    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" containerName="istio-proxy"
Oct 31 12:54:03.663799 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.663790    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39d5190-02f9-4b13-be8a-c6d4f8f2b3a1" containerName="istio-proxy"
Oct 31 12:54:03.668928 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.668910    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"
Oct 31 12:54:03.681897 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.681870    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"]
Oct 31 12:54:03.682045 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.682024    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-71bc05\""
Oct 31 12:54:03.717831 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.717735    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-71bc05-dockercfg-m74mt\""
Oct 31 12:54:03.754840 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.754804    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx\" (UID: \"a3328d1a-9f2b-42de-a5b6-35ad852a0d07\") " pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"
Oct 31 12:54:03.755293 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.755268    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcjql\" (UniqueName: \"kubernetes.io/projected/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-kube-api-access-gcjql\") pod \"payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx\" (UID: \"a3328d1a-9f2b-42de-a5b6-35ad852a0d07\") " pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"
Oct 31 12:54:03.885394 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.871991    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcjql\" (UniqueName: \"kubernetes.io/projected/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-kube-api-access-gcjql\") pod \"payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx\" (UID: \"a3328d1a-9f2b-42de-a5b6-35ad852a0d07\") " pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"
Oct 31 12:54:03.885394 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.872104    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx\" (UID: \"a3328d1a-9f2b-42de-a5b6-35ad852a0d07\") " pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"
Oct 31 12:54:03.885394 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.873320    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx\" (UID: \"a3328d1a-9f2b-42de-a5b6-35ad852a0d07\") " pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"
Oct 31 12:54:03.991439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.984259    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcjql\" (UniqueName: \"kubernetes.io/projected/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-kube-api-access-gcjql\") pod \"payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx\" (UID: \"a3328d1a-9f2b-42de-a5b6-35ad852a0d07\") " pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"
Oct 31 12:54:04.000198 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:03.998837    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"
Oct 31 12:54:04.256429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.256201    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main" containerID="cri-o://a9aff0a174dca2db750479269387a680ae5e8c838e6798eb5a0e6257653f66b0" gracePeriod=45
Oct 31 12:54:04.258833 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.258790    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main" containerID="cri-o://b2e1410854f86d8ff1f0d049e59c1342f613356f64af72ea2cb7a07d19f81d72" gracePeriod=45
Oct 31 12:54:04.455669 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.455022    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"]
Oct 31 12:54:04.504609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.504575    5663 generic.go:356] "Generic (PLEG): container finished" podID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerID="b2e1410854f86d8ff1f0d049e59c1342f613356f64af72ea2cb7a07d19f81d72" exitCode=0
Oct 31 12:54:04.504767 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.504745    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" event={"ID":"71a99b5e-57ec-41c2-9fdb-b380a0e4a152","Type":"ContainerDied","Data":"b2e1410854f86d8ff1f0d049e59c1342f613356f64af72ea2cb7a07d19f81d72"}
Oct 31 12:54:04.507737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.507668    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx" event={"ID":"a3328d1a-9f2b-42de-a5b6-35ad852a0d07","Type":"ContainerStarted","Data":"23e36e1b906d7c9da087e007b2387874a62af1922c5a8600c5604856b28417a8"}
Oct 31 12:54:04.515860 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.514572    5663 generic.go:356] "Generic (PLEG): container finished" podID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerID="a9aff0a174dca2db750479269387a680ae5e8c838e6798eb5a0e6257653f66b0" exitCode=0
Oct 31 12:54:04.515860 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.514624    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" event={"ID":"4ae07a54-ebb0-4622-b6e8-8040a947ea0f","Type":"ContainerDied","Data":"a9aff0a174dca2db750479269387a680ae5e8c838e6798eb5a0e6257653f66b0"}
Oct 31 12:54:04.551443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.550802    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:54:04.633061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.633038    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:54:04.710991 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.710955    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tls-certs\" (UniqueName: \"kubernetes.io/secret/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tls-certs\") pod \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") "
Oct 31 12:54:04.710991 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.710996    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-home\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-home\") pod \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") "
Oct 31 12:54:04.711240 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711102    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-dshm\") pod \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") "
Oct 31 12:54:04.711240 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711161    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-dshm\") pod \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") "
Oct 31 12:54:04.711240 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711199    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-model-cache\") pod \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") "
Oct 31 12:54:04.711471 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711282    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tmp-dir\") pod \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") "
Oct 31 12:54:04.711471 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711328    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-home\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-home\") pod \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") "
Oct 31 12:54:04.711471 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711371    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-home" pod "4ae07a54-ebb0-4622-b6e8-8040a947ea0f" (UID: "4ae07a54-ebb0-4622-b6e8-8040a947ea0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:04.711471 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711443    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-kube-api-access-pzm66\" (UniqueName: \"kubernetes.io/projected/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-kube-api-access-pzm66\") pod \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\" (UID: \"71a99b5e-57ec-41c2-9fdb-b380a0e4a152\") "
Oct 31 12:54:04.711686 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711474    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tmp-dir\") pod \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") "
Oct 31 12:54:04.711686 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711522    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tls-certs\") pod \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") "
Oct 31 12:54:04.711686 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711579    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-model-cache\") pod \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") "
Oct 31 12:54:04.711686 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711636    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-kube-api-access-vbscw\" (UniqueName: \"kubernetes.io/projected/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-kube-api-access-vbscw\") pod \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\" (UID: \"4ae07a54-ebb0-4622-b6e8-8040a947ea0f\") "
Oct 31 12:54:04.711686 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711612    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-model-cache" pod "71a99b5e-57ec-41c2-9fdb-b380a0e4a152" (UID: "71a99b5e-57ec-41c2-9fdb-b380a0e4a152"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:04.711955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711688    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tmp-dir" pod "71a99b5e-57ec-41c2-9fdb-b380a0e4a152" (UID: "71a99b5e-57ec-41c2-9fdb-b380a0e4a152"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:04.711955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.711825    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tmp-dir" pod "4ae07a54-ebb0-4622-b6e8-8040a947ea0f" (UID: "4ae07a54-ebb0-4622-b6e8-8040a947ea0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:04.712059 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.712008    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-home" pod "71a99b5e-57ec-41c2-9fdb-b380a0e4a152" (UID: "71a99b5e-57ec-41c2-9fdb-b380a0e4a152"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:04.712597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.712570    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:04.712723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.712623    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:04.712723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.712642    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:04.712723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.712575    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-model-cache" pod "4ae07a54-ebb0-4622-b6e8-8040a947ea0f" (UID: "4ae07a54-ebb0-4622-b6e8-8040a947ea0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:04.712723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.712698    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:04.712723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.712713    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:04.714319 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.714291    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tls-certs" pod "71a99b5e-57ec-41c2-9fdb-b380a0e4a152" (UID: "71a99b5e-57ec-41c2-9fdb-b380a0e4a152"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:04.714437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.714348    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-dshm" pod "4ae07a54-ebb0-4622-b6e8-8040a947ea0f" (UID: "4ae07a54-ebb0-4622-b6e8-8040a947ea0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:04.714437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.714402    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-kube-api-access-pzm66" pod "71a99b5e-57ec-41c2-9fdb-b380a0e4a152" (UID: "71a99b5e-57ec-41c2-9fdb-b380a0e4a152"). InnerVolumeSpecName "kube-api-access-pzm66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:04.714562 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.714433    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-kube-api-access-vbscw" pod "4ae07a54-ebb0-4622-b6e8-8040a947ea0f" (UID: "4ae07a54-ebb0-4622-b6e8-8040a947ea0f"). InnerVolumeSpecName "kube-api-access-vbscw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:04.714986 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.714961    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-dshm" pod "71a99b5e-57ec-41c2-9fdb-b380a0e4a152" (UID: "71a99b5e-57ec-41c2-9fdb-b380a0e4a152"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:04.715765 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.715746    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tls-certs" pod "4ae07a54-ebb0-4622-b6e8-8040a947ea0f" (UID: "4ae07a54-ebb0-4622-b6e8-8040a947ea0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:04.814704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.814678    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzm66\" (UniqueName: \"kubernetes.io/projected/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-kube-api-access-pzm66\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:04.814704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.814705    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:04.814912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.814716    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:04.814912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.814726    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbscw\" (UniqueName: \"kubernetes.io/projected/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-kube-api-access-vbscw\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:04.814912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.814735    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:04.814912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.814744    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ae07a54-ebb0-4622-b6e8-8040a947ea0f-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:04.814912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:04.814752    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a99b5e-57ec-41c2-9fdb-b380a0e4a152-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:05.524792 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.524755    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx" event={"ID":"a3328d1a-9f2b-42de-a5b6-35ad852a0d07","Type":"ContainerStarted","Data":"38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072"}
Oct 31 12:54:05.525207 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.524828    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"
Oct 31 12:54:05.526196 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.526173    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"
Oct 31 12:54:05.526342 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.526218    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m" event={"ID":"4ae07a54-ebb0-4622-b6e8-8040a947ea0f","Type":"ContainerDied","Data":"42f54a19183be988c1b20722a1dcff2d3c06f64a5e923e12b90082ef8de32b0d"}
Oct 31 12:54:05.526342 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.526259    5663 scope.go:122] "RemoveContainer" containerID="a9aff0a174dca2db750479269387a680ae5e8c838e6798eb5a0e6257653f66b0"
Oct 31 12:54:05.527879 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.527857    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z" event={"ID":"71a99b5e-57ec-41c2-9fdb-b380a0e4a152","Type":"ContainerDied","Data":"0aabc58fa1533fa1613eac46810d8283f4d47f25c171602f53c3616ceb0b577f"}
Oct 31 12:54:05.527956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.527877    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"
Oct 31 12:54:05.536726 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.536699    5663 scope.go:122] "RemoveContainer" containerID="b2e1410854f86d8ff1f0d049e59c1342f613356f64af72ea2cb7a07d19f81d72"
Oct 31 12:54:05.562010 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.561962    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx" podStartSLOduration=2.56194826 podStartE2EDuration="2.56194826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:05.559436718 +0000 UTC m=+1570.299862936" watchObservedRunningTime="2026-10-31 12:54:05.56194826 +0000 UTC m=+1570.302374475"
Oct 31 12:54:05.584190 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.584157    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"]
Oct 31 12:54:05.592204 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.592178    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/simulator-w1-85ce08-kserve-5766b7b6df-pzr4z"]
Oct 31 12:54:05.619275 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.619243    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"]
Oct 31 12:54:05.624522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:05.624498    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-85ce08/premium-w1-85ce08-kserve-8494f7b776-j675m"]
Oct 31 12:54:06.771987 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:06.771952    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx" podUID="97a2a833-a8a0-499d-8784-310253f03010" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:54:07.441578 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:07.441549    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" path="/var/lib/kubelet/pods/4ae07a54-ebb0-4622-b6e8-8040a947ea0f/volumes"
Oct 31 12:54:07.442065 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:07.442050    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" path="/var/lib/kubelet/pods/71a99b5e-57ec-41c2-9fdb-b380a0e4a152/volumes"
Oct 31 12:54:07.517167 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:07.517129    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"]
Oct 31 12:54:09.431222 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:09.431187    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"
Oct 31 12:54:10.448062 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:10.448033    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"
Oct 31 12:54:10.448536 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:10.448118    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"
Oct 31 12:54:10.448536 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:10.448171    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"
Oct 31 12:54:12.518456 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:12.518394    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk" podUID="ca75a2f1-e93d-47ca-a966-1bd1ad9514a5" containerName="payload-processing" containerID="cri-o://cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b" gracePeriod=25
Oct 31 12:54:12.810569 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:12.810544    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"
Oct 31 12:54:12.940907 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:12.939520    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-plugins-config-volume\") pod \"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5\" (UID: \"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5\") "
Oct 31 12:54:12.940907 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:12.939616    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-kube-api-access-dcbmm\" (UniqueName: \"kubernetes.io/projected/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-kube-api-access-dcbmm\") pod \"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5\" (UID: \"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5\") "
Oct 31 12:54:12.942764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:12.942732    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-plugins-config-volume" pod "ca75a2f1-e93d-47ca-a966-1bd1ad9514a5" (UID: "ca75a2f1-e93d-47ca-a966-1bd1ad9514a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:12.956551 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:12.956512    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-kube-api-access-dcbmm" pod "ca75a2f1-e93d-47ca-a966-1bd1ad9514a5" (UID: "ca75a2f1-e93d-47ca-a966-1bd1ad9514a5"). InnerVolumeSpecName "kube-api-access-dcbmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:13.042156 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:13.042069    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:13.042156 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:13.042103    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcbmm\" (UniqueName: \"kubernetes.io/projected/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5-kube-api-access-dcbmm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:13.643617 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:13.643579    5663 generic.go:356] "Generic (PLEG): container finished" podID="ca75a2f1-e93d-47ca-a966-1bd1ad9514a5" containerID="cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b" exitCode=0
Oct 31 12:54:13.644039 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:13.643621    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"
Oct 31 12:54:13.644039 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:13.643651    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk" event={"ID":"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5","Type":"ContainerDied","Data":"cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b"}
Oct 31 12:54:13.644039 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:13.643712    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk" event={"ID":"ca75a2f1-e93d-47ca-a966-1bd1ad9514a5","Type":"ContainerDied","Data":"3416f04c54ef9a1b97c96e166a74ffd896bb37ace52150bd481f15cdd624af3a"}
Oct 31 12:54:13.644039 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:13.643740    5663 scope.go:122] "RemoveContainer" containerID="cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b"
Oct 31 12:54:13.657021 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:13.656996    5663 scope.go:122] "RemoveContainer" containerID="cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b"
Oct 31 12:54:13.657323 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:54:13.657297    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b\": container with ID starting with cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b not found: ID does not exist" containerID="cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b"
Oct 31 12:54:13.657450 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:13.657331    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b"} err="failed to get container status \"cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b\": rpc error: code = NotFound desc = could not find container \"cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b\": container with ID starting with cf700c60863f70573eccdf185c1302081d7e6ba15a5ec65caee3811a1570939b not found: ID does not exist"
Oct 31 12:54:13.673774 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:13.673748    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"]
Oct 31 12:54:13.681709 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:13.681687    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fb2b1d-55586cff87-rzphk"]
Oct 31 12:54:15.449374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:15.448538    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca75a2f1-e93d-47ca-a966-1bd1ad9514a5" path="/var/lib/kubelet/pods/ca75a2f1-e93d-47ca-a966-1bd1ad9514a5/volumes"
Oct 31 12:54:16.085733 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.085680    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" containerID="cri-o://18f17d48a59e412d374cc66e26d78b03a3a8dd84359981900b8cc601053a9858" gracePeriod=45
Oct 31 12:54:16.128486 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.128420    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main" containerID="cri-o://29ef8e25daac3390d513112167cfd937339278b69396214040ff852d40275652" gracePeriod=45
Oct 31 12:54:16.164542 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.158914    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" containerID="cri-o://40ec43d5fd7fa0d2b5cbfd657cdb0eda260eec625a7d9654de79c7d09f5bcc7e" gracePeriod=45
Oct 31 12:54:16.216067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.216035    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": dial tcp 10.133.0.42:8000: connect: connection refused"
Oct 31 12:54:16.659457 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.658751    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"
Oct 31 12:54:16.660104 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.659860    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": dial tcp 10.133.0.47:8000: connect: connection refused"
Oct 31 12:54:16.726150 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.725994    5663 generic.go:356] "Generic (PLEG): container finished" podID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerID="40ec43d5fd7fa0d2b5cbfd657cdb0eda260eec625a7d9654de79c7d09f5bcc7e" exitCode=0
Oct 31 12:54:16.726150 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.726089    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" event={"ID":"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf","Type":"ContainerDied","Data":"40ec43d5fd7fa0d2b5cbfd657cdb0eda260eec625a7d9654de79c7d09f5bcc7e"}
Oct 31 12:54:16.729449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.728820    5663 generic.go:356] "Generic (PLEG): container finished" podID="efa39025-2185-43d3-9e91-2ae5207241c1" containerID="29ef8e25daac3390d513112167cfd937339278b69396214040ff852d40275652" exitCode=0
Oct 31 12:54:16.729449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.729002    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" event={"ID":"efa39025-2185-43d3-9e91-2ae5207241c1","Type":"ContainerDied","Data":"29ef8e25daac3390d513112167cfd937339278b69396214040ff852d40275652"}
Oct 31 12:54:16.732009 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.731878    5663 generic.go:356] "Generic (PLEG): container finished" podID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerID="18f17d48a59e412d374cc66e26d78b03a3a8dd84359981900b8cc601053a9858" exitCode=0
Oct 31 12:54:16.732009 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.731957    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" event={"ID":"113876cb-0de7-4a4c-ae74-9fcdbee19345","Type":"ContainerDied","Data":"18f17d48a59e412d374cc66e26d78b03a3a8dd84359981900b8cc601053a9858"}
Oct 31 12:54:16.771477 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.771439    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx" podUID="97a2a833-a8a0-499d-8784-310253f03010" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:54:16.780576 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.780300    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:54:16.786515 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.786490    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:54:16.813579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.813552    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:54:16.896052 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896013    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113876cb-0de7-4a4c-ae74-9fcdbee19345-kube-api-access-b4mgd\" (UniqueName: \"kubernetes.io/projected/113876cb-0de7-4a4c-ae74-9fcdbee19345-kube-api-access-b4mgd\") pod \"113876cb-0de7-4a4c-ae74-9fcdbee19345\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") "
Oct 31 12:54:16.896229 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896118    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-tmp-dir\") pod \"113876cb-0de7-4a4c-ae74-9fcdbee19345\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") "
Oct 31 12:54:16.896229 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896155    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-model-cache\") pod \"efa39025-2185-43d3-9e91-2ae5207241c1\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") "
Oct 31 12:54:16.896229 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896185    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tls-certs\") pod \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") "
Oct 31 12:54:16.896229 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896215    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-home\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-home\") pod \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") "
Oct 31 12:54:16.896404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896263    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-dshm\") pod \"efa39025-2185-43d3-9e91-2ae5207241c1\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") "
Oct 31 12:54:16.896454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896402    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tmp-dir\") pod \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") "
Oct 31 12:54:16.896530 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896457    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-dshm\") pod \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") "
Oct 31 12:54:16.896530 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896516    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-dshm\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-dshm\") pod \"113876cb-0de7-4a4c-ae74-9fcdbee19345\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") "
Oct 31 12:54:16.896634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896572    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-home\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-home\") pod \"efa39025-2185-43d3-9e91-2ae5207241c1\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") "
Oct 31 12:54:16.896701 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896663    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-kube-api-access-8jdb5\" (UniqueName: \"kubernetes.io/projected/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-kube-api-access-8jdb5\") pod \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") "
Oct 31 12:54:16.896762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896699    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-tmp-dir" pod "113876cb-0de7-4a4c-ae74-9fcdbee19345" (UID: "113876cb-0de7-4a4c-ae74-9fcdbee19345"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.896762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896708    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa39025-2185-43d3-9e91-2ae5207241c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/efa39025-2185-43d3-9e91-2ae5207241c1-tls-certs\") pod \"efa39025-2185-43d3-9e91-2ae5207241c1\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") "
Oct 31 12:54:16.896867 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896781    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-model-cache\") pod \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\" (UID: \"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf\") "
Oct 31 12:54:16.896867 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896825    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-model-cache\") pod \"113876cb-0de7-4a4c-ae74-9fcdbee19345\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") "
Oct 31 12:54:16.896867 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896860    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-home\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-home\") pod \"113876cb-0de7-4a4c-ae74-9fcdbee19345\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") "
Oct 31 12:54:16.897018 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896965    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/113876cb-0de7-4a4c-ae74-9fcdbee19345-tls-certs\" (UniqueName: \"kubernetes.io/secret/113876cb-0de7-4a4c-ae74-9fcdbee19345-tls-certs\") pod \"113876cb-0de7-4a4c-ae74-9fcdbee19345\" (UID: \"113876cb-0de7-4a4c-ae74-9fcdbee19345\") "
Oct 31 12:54:16.897018 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.896974    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-home" pod "efa39025-2185-43d3-9e91-2ae5207241c1" (UID: "efa39025-2185-43d3-9e91-2ae5207241c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.897125 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.897104    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-tmp-dir\") pod \"efa39025-2185-43d3-9e91-2ae5207241c1\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") "
Oct 31 12:54:16.897184 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.897167    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa39025-2185-43d3-9e91-2ae5207241c1-kube-api-access-pzbl9\" (UniqueName: \"kubernetes.io/projected/efa39025-2185-43d3-9e91-2ae5207241c1-kube-api-access-pzbl9\") pod \"efa39025-2185-43d3-9e91-2ae5207241c1\" (UID: \"efa39025-2185-43d3-9e91-2ae5207241c1\") "
Oct 31 12:54:16.897313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.897280    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-model-cache" pod "efa39025-2185-43d3-9e91-2ae5207241c1" (UID: "efa39025-2185-43d3-9e91-2ae5207241c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.897565 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.897541    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-tmp-dir" pod "efa39025-2185-43d3-9e91-2ae5207241c1" (UID: "efa39025-2185-43d3-9e91-2ae5207241c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.898257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.898228    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.898382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.898261    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.898382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.898275    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.898382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.898286    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:16.900120 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.900085    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-dshm" pod "113876cb-0de7-4a4c-ae74-9fcdbee19345" (UID: "113876cb-0de7-4a4c-ae74-9fcdbee19345"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.900209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.900152    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-dshm" pod "5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" (UID: "5f8e92eb-f0fa-4595-b18d-c9e78f360eaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.900427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.900406    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tmp-dir" pod "5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" (UID: "5f8e92eb-f0fa-4595-b18d-c9e78f360eaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.900691 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.900659    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/113876cb-0de7-4a4c-ae74-9fcdbee19345-tls-certs" pod "113876cb-0de7-4a4c-ae74-9fcdbee19345" (UID: "113876cb-0de7-4a4c-ae74-9fcdbee19345"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:16.900778 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.900745    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-home" pod "5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" (UID: "5f8e92eb-f0fa-4595-b18d-c9e78f360eaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.900848 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.900815    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa39025-2185-43d3-9e91-2ae5207241c1-kube-api-access-pzbl9" pod "efa39025-2185-43d3-9e91-2ae5207241c1" (UID: "efa39025-2185-43d3-9e91-2ae5207241c1"). InnerVolumeSpecName "kube-api-access-pzbl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:16.900911 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.900882    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113876cb-0de7-4a4c-ae74-9fcdbee19345-kube-api-access-b4mgd" pod "113876cb-0de7-4a4c-ae74-9fcdbee19345" (UID: "113876cb-0de7-4a4c-ae74-9fcdbee19345"). InnerVolumeSpecName "kube-api-access-b4mgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:16.900911 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.900900    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-home" pod "113876cb-0de7-4a4c-ae74-9fcdbee19345" (UID: "113876cb-0de7-4a4c-ae74-9fcdbee19345"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.901031 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.900979    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-model-cache" pod "5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" (UID: "5f8e92eb-f0fa-4595-b18d-c9e78f360eaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.901031 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.901024    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-model-cache" pod "113876cb-0de7-4a4c-ae74-9fcdbee19345" (UID: "113876cb-0de7-4a4c-ae74-9fcdbee19345"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:16.903210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.902021    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa39025-2185-43d3-9e91-2ae5207241c1-tls-certs" pod "efa39025-2185-43d3-9e91-2ae5207241c1" (UID: "efa39025-2185-43d3-9e91-2ae5207241c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:16.903210 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.902280    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-kube-api-access-8jdb5" pod "5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" (UID: "5f8e92eb-f0fa-4595-b18d-c9e78f360eaf"). InnerVolumeSpecName "kube-api-access-8jdb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:16.903549 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.903531    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tls-certs" pod "5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" (UID: "5f8e92eb-f0fa-4595-b18d-c9e78f360eaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:16.903752 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:16.903737    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-dshm" pod "efa39025-2185-43d3-9e91-2ae5207241c1" (UID: "efa39025-2185-43d3-9e91-2ae5207241c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.007843    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzbl9\" (UniqueName: \"kubernetes.io/projected/efa39025-2185-43d3-9e91-2ae5207241c1-kube-api-access-pzbl9\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.007883    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4mgd\" (UniqueName: \"kubernetes.io/projected/113876cb-0de7-4a4c-ae74-9fcdbee19345-kube-api-access-b4mgd\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.007903    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.007918    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.007933    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efa39025-2185-43d3-9e91-2ae5207241c1-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.007946    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.007960    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.007974    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.007988    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jdb5\" (UniqueName: \"kubernetes.io/projected/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-kube-api-access-8jdb5\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.008005    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efa39025-2185-43d3-9e91-2ae5207241c1-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.008019    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.008033    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.008047    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113876cb-0de7-4a4c-ae74-9fcdbee19345-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.019411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.008060    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113876cb-0de7-4a4c-ae74-9fcdbee19345-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:17.741208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.741178    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"
Oct 31 12:54:17.741714 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.741175    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf" event={"ID":"5f8e92eb-f0fa-4595-b18d-c9e78f360eaf","Type":"ContainerDied","Data":"8093ef6feaf726a16768130f9f4cbbe6ff20cda96acf0d1b7145593b0f28ebf8"}
Oct 31 12:54:17.741714 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.741304    5663 scope.go:122] "RemoveContainer" containerID="40ec43d5fd7fa0d2b5cbfd657cdb0eda260eec625a7d9654de79c7d09f5bcc7e"
Oct 31 12:54:17.742866 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.742841    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"
Oct 31 12:54:17.742966 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.742872    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm" event={"ID":"efa39025-2185-43d3-9e91-2ae5207241c1","Type":"ContainerDied","Data":"5785549ffe3b0ae34c99f6ec5c1ec7d5a483bcbd8e13dd74046901f358e15c93"}
Oct 31 12:54:17.744560 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.744542    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"
Oct 31 12:54:17.744654 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.744539    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj" event={"ID":"113876cb-0de7-4a4c-ae74-9fcdbee19345","Type":"ContainerDied","Data":"f1dd99db8c3e61b02267a328acbea76f897e03b12f66cec1286c08ba9100d4ba"}
Oct 31 12:54:17.752718 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.752701    5663 scope.go:122] "RemoveContainer" containerID="29ef8e25daac3390d513112167cfd937339278b69396214040ff852d40275652"
Oct 31 12:54:17.761779 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.761761    5663 scope.go:122] "RemoveContainer" containerID="18f17d48a59e412d374cc66e26d78b03a3a8dd84359981900b8cc601053a9858"
Oct 31 12:54:17.790409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.790388    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"]
Oct 31 12:54:17.797136 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.797107    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/distinct-w3-b218d3-kserve-7bc99557cd-pcpxj"]
Oct 31 12:54:17.886379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.869948    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"]
Oct 31 12:54:17.886379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.876383    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/simulator-w3-b218d3-kserve-658d9b597d-lt2bf"]
Oct 31 12:54:17.886379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.883703    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"]
Oct 31 12:54:17.908182 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:17.906289    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b218d3/premium-w3-b218d3-kserve-5c9776df9-dqtmm"]
Oct 31 12:54:19.442585 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:19.442552    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" path="/var/lib/kubelet/pods/113876cb-0de7-4a4c-ae74-9fcdbee19345/volumes"
Oct 31 12:54:19.443095 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:19.443080    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" path="/var/lib/kubelet/pods/5f8e92eb-f0fa-4595-b18d-c9e78f360eaf/volumes"
Oct 31 12:54:19.443534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:19.443519    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" path="/var/lib/kubelet/pods/efa39025-2185-43d3-9e91-2ae5207241c1/volumes"
Oct 31 12:54:20.328133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:20.328096    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"
Oct 31 12:54:20.351000 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:20.350964    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a2a833-a8a0-499d-8784-310253f03010-kube-api-access-bfrw6\" (UniqueName: \"kubernetes.io/projected/97a2a833-a8a0-499d-8784-310253f03010-kube-api-access-bfrw6\") pod \"97a2a833-a8a0-499d-8784-310253f03010\" (UID: \"97a2a833-a8a0-499d-8784-310253f03010\") "
Oct 31 12:54:20.351192 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:20.351051    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a2a833-a8a0-499d-8784-310253f03010-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a2a833-a8a0-499d-8784-310253f03010-plugins-config-volume\") pod \"97a2a833-a8a0-499d-8784-310253f03010\" (UID: \"97a2a833-a8a0-499d-8784-310253f03010\") "
Oct 31 12:54:20.351465 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:20.351441    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a2a833-a8a0-499d-8784-310253f03010-plugins-config-volume" pod "97a2a833-a8a0-499d-8784-310253f03010" (UID: "97a2a833-a8a0-499d-8784-310253f03010"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:20.351966 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:20.351939    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a2a833-a8a0-499d-8784-310253f03010-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:20.353226 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:20.353196    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a2a833-a8a0-499d-8784-310253f03010-kube-api-access-bfrw6" pod "97a2a833-a8a0-499d-8784-310253f03010" (UID: "97a2a833-a8a0-499d-8784-310253f03010"). InnerVolumeSpecName "kube-api-access-bfrw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:20.453516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:20.453482    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfrw6\" (UniqueName: \"kubernetes.io/projected/97a2a833-a8a0-499d-8784-310253f03010-kube-api-access-bfrw6\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:20.574680 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:20.574592    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"]
Oct 31 12:54:20.574905 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:20.574868    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct" podUID="fc521967-8ba8-4074-bf02-9e7ebf169601" containerName="maas-api" containerID="cri-o://88c227e38b50c709c24ad55613bef3d2b64ed7c9ad206986ec2a594cbc93640f" gracePeriod=30
Oct 31 12:54:21.302067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.301742    5663 generic.go:356] "Generic (PLEG): container finished" podID="fc521967-8ba8-4074-bf02-9e7ebf169601" containerID="88c227e38b50c709c24ad55613bef3d2b64ed7c9ad206986ec2a594cbc93640f" exitCode=0
Oct 31 12:54:21.302067 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.301867    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct" event={"ID":"fc521967-8ba8-4074-bf02-9e7ebf169601","Type":"ContainerDied","Data":"88c227e38b50c709c24ad55613bef3d2b64ed7c9ad206986ec2a594cbc93640f"}
Oct 31 12:54:21.307817 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.303997    5663 generic.go:356] "Generic (PLEG): container finished" podID="97a2a833-a8a0-499d-8784-310253f03010" containerID="77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011" exitCode=0
Oct 31 12:54:21.307817 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.304300    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx" event={"ID":"97a2a833-a8a0-499d-8784-310253f03010","Type":"ContainerDied","Data":"77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011"}
Oct 31 12:54:21.307817 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.304373    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx" event={"ID":"97a2a833-a8a0-499d-8784-310253f03010","Type":"ContainerDied","Data":"d4777435abd8a32d0df77e640e774ed437669052e86b1a3e81076ff0d806d0c1"}
Oct 31 12:54:21.307817 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.304412    5663 scope.go:122] "RemoveContainer" containerID="77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011"
Oct 31 12:54:21.307817 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.304571    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"
Oct 31 12:54:21.328929 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.328907    5663 scope.go:122] "RemoveContainer" containerID="77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011"
Oct 31 12:54:21.329201 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:54:21.329179    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011\": container with ID starting with 77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011 not found: ID does not exist" containerID="77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011"
Oct 31 12:54:21.329246 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.329211    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011"} err="failed to get container status \"77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011\": rpc error: code = NotFound desc = could not find container \"77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011\": container with ID starting with 77fb417bc6cf1fa18f51ca45b6f04af299db2055385ea8d5f19a290a951a8011 not found: ID does not exist"
Oct 31 12:54:21.330415 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.330391    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:54:21.342851 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.342827    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"]
Oct 31 12:54:21.349338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.349317    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-85ce08-f7bdb5c75-62gtx"]
Oct 31 12:54:21.380887 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.380859    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-metrics-certs\") pod \"fc521967-8ba8-4074-bf02-9e7ebf169601\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") "
Oct 31 12:54:21.381014 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.380895    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc521967-8ba8-4074-bf02-9e7ebf169601-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc521967-8ba8-4074-bf02-9e7ebf169601-openshift-service-ca\") pod \"fc521967-8ba8-4074-bf02-9e7ebf169601\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") "
Oct 31 12:54:21.381077 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.381028    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc521967-8ba8-4074-bf02-9e7ebf169601-kube-api-access-8c6lh\" (UniqueName: \"kubernetes.io/projected/fc521967-8ba8-4074-bf02-9e7ebf169601-kube-api-access-8c6lh\") pod \"fc521967-8ba8-4074-bf02-9e7ebf169601\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") "
Oct 31 12:54:21.381180 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.381162    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-maas-api-tls\") pod \"fc521967-8ba8-4074-bf02-9e7ebf169601\" (UID: \"fc521967-8ba8-4074-bf02-9e7ebf169601\") "
Oct 31 12:54:21.381343 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.381318    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc521967-8ba8-4074-bf02-9e7ebf169601-openshift-service-ca" pod "fc521967-8ba8-4074-bf02-9e7ebf169601" (UID: "fc521967-8ba8-4074-bf02-9e7ebf169601"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:21.382311 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.382071    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc521967-8ba8-4074-bf02-9e7ebf169601-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:21.383748 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.383720    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-maas-api-tls" pod "fc521967-8ba8-4074-bf02-9e7ebf169601" (UID: "fc521967-8ba8-4074-bf02-9e7ebf169601"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:21.383961 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.383936    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc521967-8ba8-4074-bf02-9e7ebf169601-kube-api-access-8c6lh" pod "fc521967-8ba8-4074-bf02-9e7ebf169601" (UID: "fc521967-8ba8-4074-bf02-9e7ebf169601"). InnerVolumeSpecName "kube-api-access-8c6lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:21.384101 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.384081    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-metrics-certs" pod "fc521967-8ba8-4074-bf02-9e7ebf169601" (UID: "fc521967-8ba8-4074-bf02-9e7ebf169601"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:21.440570 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.440538    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a2a833-a8a0-499d-8784-310253f03010" path="/var/lib/kubelet/pods/97a2a833-a8a0-499d-8784-310253f03010/volumes"
Oct 31 12:54:21.483517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.483477    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c6lh\" (UniqueName: \"kubernetes.io/projected/fc521967-8ba8-4074-bf02-9e7ebf169601-kube-api-access-8c6lh\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:21.483517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.483507    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:21.483517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.483518    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc521967-8ba8-4074-bf02-9e7ebf169601-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:21.535552 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.535477    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"]
Oct 31 12:54:21.536148 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536133    5663 cpu_manager.go:414] "containerMap: removing container" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main"
Oct 31 12:54:21.536148 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536148    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536161    5663 cpu_manager.go:414] "containerMap: removing container" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536166    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536177    5663 cpu_manager.go:414] "containerMap: removing container" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536182    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536190    5663 cpu_manager.go:414] "containerMap: removing container" podUID="97a2a833-a8a0-499d-8784-310253f03010" containerName="payload-pre-processing"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536195    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a2a833-a8a0-499d-8784-310253f03010" containerName="payload-pre-processing"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536204    5663 cpu_manager.go:414] "containerMap: removing container" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536208    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536231    5663 cpu_manager.go:414] "containerMap: removing container" podUID="ca75a2f1-e93d-47ca-a966-1bd1ad9514a5" containerName="payload-processing"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536235    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca75a2f1-e93d-47ca-a966-1bd1ad9514a5" containerName="payload-processing"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536244    5663 cpu_manager.go:414] "containerMap: removing container" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536248    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536255    5663 cpu_manager.go:414] "containerMap: removing container" podUID="fc521967-8ba8-4074-bf02-9e7ebf169601" containerName="maas-api"
Oct 31 12:54:21.536257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536259    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc521967-8ba8-4074-bf02-9e7ebf169601" containerName="maas-api"
Oct 31 12:54:21.536724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536391    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca75a2f1-e93d-47ca-a966-1bd1ad9514a5" containerName="payload-processing"
Oct 31 12:54:21.536724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536406    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a99b5e-57ec-41c2-9fdb-b380a0e4a152" containerName="main"
Oct 31 12:54:21.536724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536416    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc521967-8ba8-4074-bf02-9e7ebf169601" containerName="maas-api"
Oct 31 12:54:21.536724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536436    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae07a54-ebb0-4622-b6e8-8040a947ea0f" containerName="main"
Oct 31 12:54:21.536724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536446    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f8e92eb-f0fa-4595-b18d-c9e78f360eaf" containerName="main"
Oct 31 12:54:21.536724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536452    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa39025-2185-43d3-9e91-2ae5207241c1" containerName="main"
Oct 31 12:54:21.536724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536458    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a2a833-a8a0-499d-8784-310253f03010" containerName="payload-pre-processing"
Oct 31 12:54:21.536724 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.536464    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="113876cb-0de7-4a4c-ae74-9fcdbee19345" containerName="main"
Oct 31 12:54:21.541335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.541316    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.549081 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.549057    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-71bc05-serving-cert\""
Oct 31 12:54:21.550798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.550776    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"]
Oct 31 12:54:21.585317 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.585283    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-metrics-certs\") pod \"maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.585468 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.585423    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f09a69d-29df-4586-8836-49bdc029a99b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.585611 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.585592    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqvb\" (UniqueName: \"kubernetes.io/projected/1f09a69d-29df-4586-8836-49bdc029a99b-kube-api-access-vsqvb\") pod \"maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.585842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.585826    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-maas-api-tls\") pod \"maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.686808 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.686779    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-maas-api-tls\") pod \"maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.686997 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.686848    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-metrics-certs\") pod \"maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.686997 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.686889    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f09a69d-29df-4586-8836-49bdc029a99b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.686997 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.686971    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqvb\" (UniqueName: \"kubernetes.io/projected/1f09a69d-29df-4586-8836-49bdc029a99b-kube-api-access-vsqvb\") pod \"maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.687810 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.687778    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f09a69d-29df-4586-8836-49bdc029a99b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.689398 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.689346    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-maas-api-tls\") pod \"maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.689492 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.689457    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-metrics-certs\") pod \"maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.699124 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.699101    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqvb\" (UniqueName: \"kubernetes.io/projected/1f09a69d-29df-4586-8836-49bdc029a99b-kube-api-access-vsqvb\") pod \"maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:21.861921 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:21.861888    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:22.218924 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:54:22.218892    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f09a69d_29df_4586_8836_49bdc029a99b.slice/crio-ca672e7cf112d8770ef230254886c5cb7cc9aea92076ae3bcc40a159466d5b45 WatchSource:0}: Error finding container ca672e7cf112d8770ef230254886c5cb7cc9aea92076ae3bcc40a159466d5b45: Status 404 returned error can't find the container with id ca672e7cf112d8770ef230254886c5cb7cc9aea92076ae3bcc40a159466d5b45
Oct 31 12:54:22.220188 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:22.220161    5663 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:54:22.333380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:22.332861    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"]
Oct 31 12:54:22.398912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:22.398874    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb" event={"ID":"1f09a69d-29df-4586-8836-49bdc029a99b","Type":"ContainerStarted","Data":"ca672e7cf112d8770ef230254886c5cb7cc9aea92076ae3bcc40a159466d5b45"}
Oct 31 12:54:22.413435 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:22.413331    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct" event={"ID":"fc521967-8ba8-4074-bf02-9e7ebf169601","Type":"ContainerDied","Data":"7cf04b02a261779a6368d816e50864b60e313145f8c08a38d526ebaef4526f27"}
Oct 31 12:54:22.413435 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:22.413428    5663 scope.go:122] "RemoveContainer" containerID="88c227e38b50c709c24ad55613bef3d2b64ed7c9ad206986ec2a594cbc93640f"
Oct 31 12:54:22.419477 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:22.419444    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"
Oct 31 12:54:22.517617 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:22.516336    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"]
Oct 31 12:54:22.533041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:22.533001    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fb2b1d-7475594df5-lwhct"]
Oct 31 12:54:22.568180 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:54:22.568141    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:54:23.440898 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:23.440864    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc521967-8ba8-4074-bf02-9e7ebf169601" path="/var/lib/kubelet/pods/fc521967-8ba8-4074-bf02-9e7ebf169601/volumes"
Oct 31 12:54:23.469766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:23.469726    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb" event={"ID":"1f09a69d-29df-4586-8836-49bdc029a99b","Type":"ContainerStarted","Data":"6615b6491bf939288f4b3d178572e009483523234262be9f6bc92cc8cf3e452e"}
Oct 31 12:54:23.469952 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:23.469927    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:23.490016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:23.489962    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb" podStartSLOduration=2.005919755 podStartE2EDuration="2.489949846s" totalImagesPullingTime="484.030091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:23.488648008 +0000 UTC m=+1588.229074224" watchObservedRunningTime="2026-10-31 12:54:23.489949846 +0000 UTC m=+1588.230376073"
Oct 31 12:54:29.044340 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:29.043438    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"]
Oct 31 12:54:29.044340 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:29.043652    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy" containerID="cri-o://ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3" gracePeriod=30
Oct 31 12:54:29.052364 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:29.051969    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"]
Oct 31 12:54:29.052364 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:29.052270    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy" containerID="cri-o://e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372" gracePeriod=30
Oct 31 12:54:29.501431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:29.500057    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:54:31.832671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:31.831183    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:54:31.832671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:31.832255    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:54:32.887827 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:32.887790    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"]
Oct 31 12:54:32.888716 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:32.888674    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" podUID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" containerName="istio-proxy" containerID="cri-o://d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc" gracePeriod=30
Oct 31 12:54:34.011936 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.011898    5663 patch_prober.go:29] interesting pod/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:54:34.012336 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.011955    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:54:34.341486 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.341458    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:54:34.385224 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.385197    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:54:34.440472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.440424    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-data\") pod \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") "
Oct 31 12:54:34.440676 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.440557    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-token\" (UniqueName: \"kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-token\") pod \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") "
Oct 31 12:54:34.440676 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.440636    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-podinfo\") pod \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") "
Oct 31 12:54:34.440848 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.440697    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-data" pod "cf969552-deff-4db0-91e7-45ff2b8a9c2b" (UID: "cf969552-deff-4db0-91e7-45ff2b8a9c2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.440848 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.440715    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-certs\") pod \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") "
Oct 31 12:54:34.440848 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.440763    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-token\" (UniqueName: \"kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-token\") pod \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") "
Oct 31 12:54:34.440848 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.440798    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-podinfo\") pod \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") "
Oct 31 12:54:34.440848 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.440819    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-credential-socket\") pod \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") "
Oct 31 12:54:34.441123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.440852    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-kube-api-access-nt8kh\" (UniqueName: \"kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-kube-api-access-nt8kh\") pod \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") "
Oct 31 12:54:34.441123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.440972    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-socket\") pod \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") "
Oct 31 12:54:34.441123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.440975    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-certs" pod "cf969552-deff-4db0-91e7-45ff2b8a9c2b" (UID: "cf969552-deff-4db0-91e7-45ff2b8a9c2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.441123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.441026    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-kube-api-access-tnw7g\" (UniqueName: \"kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-kube-api-access-tnw7g\") pod \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") "
Oct 31 12:54:34.441123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.441078    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istiod-ca-cert\") pod \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") "
Oct 31 12:54:34.441123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.441122    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-socket\") pod \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") "
Oct 31 12:54:34.441589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.441151    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-credential-socket\") pod \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") "
Oct 31 12:54:34.441589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.441209    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-credential-socket" pod "d0c999a6-2915-4a7b-8f8f-2862bf0f880e" (UID: "d0c999a6-2915-4a7b-8f8f-2862bf0f880e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.441589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.441462    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-socket" pod "cf969552-deff-4db0-91e7-45ff2b8a9c2b" (UID: "cf969552-deff-4db0-91e7-45ff2b8a9c2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.441843 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.441713    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istiod-ca-cert" pod "cf969552-deff-4db0-91e7-45ff2b8a9c2b" (UID: "cf969552-deff-4db0-91e7-45ff2b8a9c2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:34.442439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.442399    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-socket" pod "d0c999a6-2915-4a7b-8f8f-2862bf0f880e" (UID: "d0c999a6-2915-4a7b-8f8f-2862bf0f880e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.443104 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.442620    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istiod-ca-cert\") pod \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") "
Oct 31 12:54:34.443314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.442816    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-credential-socket" pod "cf969552-deff-4db0-91e7-45ff2b8a9c2b" (UID: "cf969552-deff-4db0-91e7-45ff2b8a9c2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.443314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.443078    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istiod-ca-cert" pod "d0c999a6-2915-4a7b-8f8f-2862bf0f880e" (UID: "d0c999a6-2915-4a7b-8f8f-2862bf0f880e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:34.443527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.443509    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-envoy\") pod \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") "
Oct 31 12:54:34.443935 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.443909    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-data\") pod \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") "
Oct 31 12:54:34.444051 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.444033    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-envoy\") pod \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\" (UID: \"cf969552-deff-4db0-91e7-45ff2b8a9c2b\") "
Oct 31 12:54:34.444133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.444114    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-certs\") pod \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\" (UID: \"d0c999a6-2915-4a7b-8f8f-2862bf0f880e\") "
Oct 31 12:54:34.444990 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.444966    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.445096 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.444996    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.445096 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.445012    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.445096 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.445025    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.445096 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.445037    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.445096 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.445052    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.445096 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.445066    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.445096 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.445078    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.446367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.445368    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-certs" pod "d0c999a6-2915-4a7b-8f8f-2862bf0f880e" (UID: "d0c999a6-2915-4a7b-8f8f-2862bf0f880e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.446367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.445827    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-token" pod "d0c999a6-2915-4a7b-8f8f-2862bf0f880e" (UID: "d0c999a6-2915-4a7b-8f8f-2862bf0f880e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:34.446367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.445863    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-kube-api-access-tnw7g" pod "cf969552-deff-4db0-91e7-45ff2b8a9c2b" (UID: "cf969552-deff-4db0-91e7-45ff2b8a9c2b"). InnerVolumeSpecName "kube-api-access-tnw7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:34.446367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.445874    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-data" pod "d0c999a6-2915-4a7b-8f8f-2862bf0f880e" (UID: "d0c999a6-2915-4a7b-8f8f-2862bf0f880e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.446367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.445904    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-kube-api-access-nt8kh" pod "d0c999a6-2915-4a7b-8f8f-2862bf0f880e" (UID: "d0c999a6-2915-4a7b-8f8f-2862bf0f880e"). InnerVolumeSpecName "kube-api-access-nt8kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:34.446367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.446225    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-podinfo" pod "d0c999a6-2915-4a7b-8f8f-2862bf0f880e" (UID: "d0c999a6-2915-4a7b-8f8f-2862bf0f880e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:54:34.446367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.446309    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-envoy" pod "d0c999a6-2915-4a7b-8f8f-2862bf0f880e" (UID: "d0c999a6-2915-4a7b-8f8f-2862bf0f880e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.446367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.446333    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-token" pod "cf969552-deff-4db0-91e7-45ff2b8a9c2b" (UID: "cf969552-deff-4db0-91e7-45ff2b8a9c2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:34.446367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.446329    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-podinfo" pod "cf969552-deff-4db0-91e7-45ff2b8a9c2b" (UID: "cf969552-deff-4db0-91e7-45ff2b8a9c2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:54:34.447606 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.447580    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-envoy" pod "cf969552-deff-4db0-91e7-45ff2b8a9c2b" (UID: "cf969552-deff-4db0-91e7-45ff2b8a9c2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:34.567042 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.566953    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.567042 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.566985    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.567042 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.566998    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.567042 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.567010    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.567042 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.567023    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.567042 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.567036    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.567570 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.567048    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.567570 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.567061    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.567570 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.567075    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt8kh\" (UniqueName: \"kubernetes.io/projected/d0c999a6-2915-4a7b-8f8f-2862bf0f880e-kube-api-access-nt8kh\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.567570 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.567087    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnw7g\" (UniqueName: \"kubernetes.io/projected/cf969552-deff-4db0-91e7-45ff2b8a9c2b-kube-api-access-tnw7g\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:34.718996 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.718959    5663 generic.go:356] "Generic (PLEG): container finished" podID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerID="e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372" exitCode=0
Oct 31 12:54:34.719190 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.719019    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"
Oct 31 12:54:34.719190 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.719035    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" event={"ID":"d0c999a6-2915-4a7b-8f8f-2862bf0f880e","Type":"ContainerDied","Data":"e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372"}
Oct 31 12:54:34.719190 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.719088    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw" event={"ID":"d0c999a6-2915-4a7b-8f8f-2862bf0f880e","Type":"ContainerDied","Data":"7d55b8ea81502b2b91314713a15cc03d8d60f172fb118de8c60fed3925a60857"}
Oct 31 12:54:34.719190 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.719116    5663 scope.go:122] "RemoveContainer" containerID="e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372"
Oct 31 12:54:34.720691 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.720662    5663 generic.go:356] "Generic (PLEG): container finished" podID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerID="ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3" exitCode=0
Oct 31 12:54:34.720824 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.720705    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"
Oct 31 12:54:34.720824 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.720709    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" event={"ID":"cf969552-deff-4db0-91e7-45ff2b8a9c2b","Type":"ContainerDied","Data":"ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3"}
Oct 31 12:54:34.720824 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.720753    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx" event={"ID":"cf969552-deff-4db0-91e7-45ff2b8a9c2b","Type":"ContainerDied","Data":"59f91e20b6a26b7c1feaeb5710a0d13d8b7a798146b09ce058642bc8f95e4126"}
Oct 31 12:54:34.731271 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.731236    5663 scope.go:122] "RemoveContainer" containerID="e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372"
Oct 31 12:54:34.731614 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:54:34.731589    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372\": container with ID starting with e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372 not found: ID does not exist" containerID="e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372"
Oct 31 12:54:34.731715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.731622    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372"} err="failed to get container status \"e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372\": rpc error: code = NotFound desc = could not find container \"e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372\": container with ID starting with e0a4258fdeb49d2abceb6c97713202a1a77503799e7e5f504ed0a27ba9b4a372 not found: ID does not exist"
Oct 31 12:54:34.731715 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.731639    5663 scope.go:122] "RemoveContainer" containerID="ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3"
Oct 31 12:54:34.741940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.741920    5663 scope.go:122] "RemoveContainer" containerID="ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3"
Oct 31 12:54:34.742343 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:54:34.742320    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3\": container with ID starting with ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3 not found: ID does not exist" containerID="ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3"
Oct 31 12:54:34.742437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.742385    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3"} err="failed to get container status \"ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3\": rpc error: code = NotFound desc = could not find container \"ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3\": container with ID starting with ae3fc43df9c0edadacc746e04df7a6e67341b5667aaa35291ce976814f4eead3 not found: ID does not exist"
Oct 31 12:54:34.779719 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.779687    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"]
Oct 31 12:54:34.786596 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.786564    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-kbklx"]
Oct 31 12:54:34.816169 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.816134    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"]
Oct 31 12:54:34.835278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:34.835171    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85ce08-openshift-default-77d6458845-vzgjw"]
Oct 31 12:54:35.449514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:35.449476    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" path="/var/lib/kubelet/pods/cf969552-deff-4db0-91e7-45ff2b8a9c2b/volumes"
Oct 31 12:54:35.450029 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:35.450009    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" path="/var/lib/kubelet/pods/d0c999a6-2915-4a7b-8f8f-2862bf0f880e/volumes"
Oct 31 12:54:38.282936 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.282908    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:54:38.348665 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.348627    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-certs\") pod \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") "
Oct 31 12:54:38.348857 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.348744    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-data\") pod \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") "
Oct 31 12:54:38.348857 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.348818    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-socket\") pod \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") "
Oct 31 12:54:38.348857 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.348847    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-kube-api-access-xll87\" (UniqueName: \"kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-kube-api-access-xll87\") pod \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") "
Oct 31 12:54:38.349037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.348890    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istiod-ca-cert\") pod \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") "
Oct 31 12:54:38.349037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.348954    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-certs" pod "5855c115-9e7a-46b0-89e8-d7ec52b1719b" (UID: "5855c115-9e7a-46b0-89e8-d7ec52b1719b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.349037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.348965    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-token\" (UniqueName: \"kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-token\") pod \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") "
Oct 31 12:54:38.349175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.349082    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-envoy\") pod \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") "
Oct 31 12:54:38.349175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.349149    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-podinfo\") pod \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") "
Oct 31 12:54:38.349293 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.349233    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-credential-socket\") pod \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\" (UID: \"5855c115-9e7a-46b0-89e8-d7ec52b1719b\") "
Oct 31 12:54:38.349756 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.349607    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-socket" pod "5855c115-9e7a-46b0-89e8-d7ec52b1719b" (UID: "5855c115-9e7a-46b0-89e8-d7ec52b1719b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.349756 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.349696    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istiod-ca-cert" pod "5855c115-9e7a-46b0-89e8-d7ec52b1719b" (UID: "5855c115-9e7a-46b0-89e8-d7ec52b1719b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:38.350013 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.349993    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.350089 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.350015    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.350089 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.350024    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.351478 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.351310    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-credential-socket" pod "5855c115-9e7a-46b0-89e8-d7ec52b1719b" (UID: "5855c115-9e7a-46b0-89e8-d7ec52b1719b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.351478 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.351499    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-data" pod "5855c115-9e7a-46b0-89e8-d7ec52b1719b" (UID: "5855c115-9e7a-46b0-89e8-d7ec52b1719b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.351914 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.351890    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-podinfo" pod "5855c115-9e7a-46b0-89e8-d7ec52b1719b" (UID: "5855c115-9e7a-46b0-89e8-d7ec52b1719b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:54:38.353070 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.353045    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-envoy" pod "5855c115-9e7a-46b0-89e8-d7ec52b1719b" (UID: "5855c115-9e7a-46b0-89e8-d7ec52b1719b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:54:38.353175 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.353080    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-token" pod "5855c115-9e7a-46b0-89e8-d7ec52b1719b" (UID: "5855c115-9e7a-46b0-89e8-d7ec52b1719b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:38.353834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.353817    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-kube-api-access-xll87" pod "5855c115-9e7a-46b0-89e8-d7ec52b1719b" (UID: "5855c115-9e7a-46b0-89e8-d7ec52b1719b"). InnerVolumeSpecName "kube-api-access-xll87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:38.531445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.504998    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.531445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.505037    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.531445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.505053    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.531445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.505068    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xll87\" (UniqueName: \"kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-kube-api-access-xll87\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.531445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.505082    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.531445 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.505096    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5855c115-9e7a-46b0-89e8-d7ec52b1719b-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:38.772367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.772321    5663 generic.go:356] "Generic (PLEG): container finished" podID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" containerID="d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc" exitCode=0
Oct 31 12:54:38.772569 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.772406    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"
Oct 31 12:54:38.772569 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.772409    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" event={"ID":"5855c115-9e7a-46b0-89e8-d7ec52b1719b","Type":"ContainerDied","Data":"d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc"}
Oct 31 12:54:38.772569 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.772475    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2" event={"ID":"5855c115-9e7a-46b0-89e8-d7ec52b1719b","Type":"ContainerDied","Data":"19c6a1457a10a682025a254b2ca3d1528dc87ac6c4294339719a11fb18db9adf"}
Oct 31 12:54:38.772569 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.772500    5663 scope.go:122] "RemoveContainer" containerID="d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc"
Oct 31 12:54:38.787423 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.787397    5663 scope.go:122] "RemoveContainer" containerID="d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc"
Oct 31 12:54:38.787803 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:54:38.787779    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc\": container with ID starting with d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc not found: ID does not exist" containerID="d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc"
Oct 31 12:54:38.787885 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.787811    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc"} err="failed to get container status \"d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc\": rpc error: code = NotFound desc = could not find container \"d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc\": container with ID starting with d1a02260b9542952813dce53412585f8025aeb5629cff1a6bdc1061b0ccd38dc not found: ID does not exist"
Oct 31 12:54:38.806072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.806036    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"]
Oct 31 12:54:38.810166 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:38.810139    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb2b1d-openshift-default-7649dffb69-d84x2"]
Oct 31 12:54:39.553027 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:39.552988    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" path="/var/lib/kubelet/pods/5855c115-9e7a-46b0-89e8-d7ec52b1719b/volumes"
Oct 31 12:54:43.623077 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:43.623037    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"]
Oct 31 12:54:48.623439 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:48.623394    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg" podUID="aca29716-d6b0-4d20-8488-556d6018aefe" containerName="payload-processing" containerID="cri-o://c0c0ac0b52b5a1e37bb1f00ba780afd9518beb9673560af1a7dede74a21e0899" gracePeriod=25
Oct 31 12:54:48.893220 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:48.893131    5663 generic.go:356] "Generic (PLEG): container finished" podID="aca29716-d6b0-4d20-8488-556d6018aefe" containerID="c0c0ac0b52b5a1e37bb1f00ba780afd9518beb9673560af1a7dede74a21e0899" exitCode=0
Oct 31 12:54:48.893418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:48.893216    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg" event={"ID":"aca29716-d6b0-4d20-8488-556d6018aefe","Type":"ContainerDied","Data":"c0c0ac0b52b5a1e37bb1f00ba780afd9518beb9673560af1a7dede74a21e0899"}
Oct 31 12:54:48.990418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:48.986441    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"
Oct 31 12:54:49.116333 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.116293    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca29716-d6b0-4d20-8488-556d6018aefe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca29716-d6b0-4d20-8488-556d6018aefe-plugins-config-volume\") pod \"aca29716-d6b0-4d20-8488-556d6018aefe\" (UID: \"aca29716-d6b0-4d20-8488-556d6018aefe\") "
Oct 31 12:54:49.121964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.116348    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca29716-d6b0-4d20-8488-556d6018aefe-kube-api-access-8hnm6\" (UniqueName: \"kubernetes.io/projected/aca29716-d6b0-4d20-8488-556d6018aefe-kube-api-access-8hnm6\") pod \"aca29716-d6b0-4d20-8488-556d6018aefe\" (UID: \"aca29716-d6b0-4d20-8488-556d6018aefe\") "
Oct 31 12:54:49.121964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.116775    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca29716-d6b0-4d20-8488-556d6018aefe-plugins-config-volume" pod "aca29716-d6b0-4d20-8488-556d6018aefe" (UID: "aca29716-d6b0-4d20-8488-556d6018aefe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:49.121964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.117101    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aca29716-d6b0-4d20-8488-556d6018aefe-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:49.126282 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.126252    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca29716-d6b0-4d20-8488-556d6018aefe-kube-api-access-8hnm6" pod "aca29716-d6b0-4d20-8488-556d6018aefe" (UID: "aca29716-d6b0-4d20-8488-556d6018aefe"). InnerVolumeSpecName "kube-api-access-8hnm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:49.218149 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.218067    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hnm6\" (UniqueName: \"kubernetes.io/projected/aca29716-d6b0-4d20-8488-556d6018aefe-kube-api-access-8hnm6\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:49.480512 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.480422    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"]
Oct 31 12:54:49.480723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.480675    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz" podUID="fa58022c-031e-45fb-9f00-d188fca88d48" containerName="maas-api" containerID="cri-o://37743c0919a42cdddaad2fb2c29b75836a06c217630baaa001cef79ab77b1450" gracePeriod=30
Oct 31 12:54:49.979126 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.971166    5663 generic.go:356] "Generic (PLEG): container finished" podID="fa58022c-031e-45fb-9f00-d188fca88d48" containerID="37743c0919a42cdddaad2fb2c29b75836a06c217630baaa001cef79ab77b1450" exitCode=0
Oct 31 12:54:49.979126 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.971463    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz" event={"ID":"fa58022c-031e-45fb-9f00-d188fca88d48","Type":"ContainerDied","Data":"37743c0919a42cdddaad2fb2c29b75836a06c217630baaa001cef79ab77b1450"}
Oct 31 12:54:49.979126 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.976653    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg" event={"ID":"aca29716-d6b0-4d20-8488-556d6018aefe","Type":"ContainerDied","Data":"77c7cbca1bfd7560ce61f4c0651a631d792f4a3f8aab89063b7bd47e85fe467a"}
Oct 31 12:54:49.979126 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.976721    5663 scope.go:122] "RemoveContainer" containerID="c0c0ac0b52b5a1e37bb1f00ba780afd9518beb9673560af1a7dede74a21e0899"
Oct 31 12:54:49.979126 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:49.976906    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"
Oct 31 12:54:50.141487 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.134540    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"]
Oct 31 12:54:50.188993 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.188293    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d4dd68-55d669b4c9-ml8bg"]
Oct 31 12:54:50.264019 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.263993    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:54:50.272047 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.272018    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa58022c-031e-45fb-9f00-d188fca88d48-kube-api-access-wb2nm\" (UniqueName: \"kubernetes.io/projected/fa58022c-031e-45fb-9f00-d188fca88d48-kube-api-access-wb2nm\") pod \"fa58022c-031e-45fb-9f00-d188fca88d48\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") "
Oct 31 12:54:50.272185 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.272129    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-maas-api-tls\") pod \"fa58022c-031e-45fb-9f00-d188fca88d48\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") "
Oct 31 12:54:50.272239 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.272228    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-metrics-certs\") pod \"fa58022c-031e-45fb-9f00-d188fca88d48\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") "
Oct 31 12:54:50.272402 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.272384    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa58022c-031e-45fb-9f00-d188fca88d48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa58022c-031e-45fb-9f00-d188fca88d48-openshift-service-ca\") pod \"fa58022c-031e-45fb-9f00-d188fca88d48\" (UID: \"fa58022c-031e-45fb-9f00-d188fca88d48\") "
Oct 31 12:54:50.272788 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.272761    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa58022c-031e-45fb-9f00-d188fca88d48-openshift-service-ca" pod "fa58022c-031e-45fb-9f00-d188fca88d48" (UID: "fa58022c-031e-45fb-9f00-d188fca88d48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:54:50.272986 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.272964    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa58022c-031e-45fb-9f00-d188fca88d48-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:50.274476 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.274448    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-maas-api-tls" pod "fa58022c-031e-45fb-9f00-d188fca88d48" (UID: "fa58022c-031e-45fb-9f00-d188fca88d48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:50.274702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.274479    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-metrics-certs" pod "fa58022c-031e-45fb-9f00-d188fca88d48" (UID: "fa58022c-031e-45fb-9f00-d188fca88d48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:54:50.274702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.274524    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa58022c-031e-45fb-9f00-d188fca88d48-kube-api-access-wb2nm" pod "fa58022c-031e-45fb-9f00-d188fca88d48" (UID: "fa58022c-031e-45fb-9f00-d188fca88d48"). InnerVolumeSpecName "kube-api-access-wb2nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:54:50.374305 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.374267    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:50.374305 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.374300    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb2nm\" (UniqueName: \"kubernetes.io/projected/fa58022c-031e-45fb-9f00-d188fca88d48-kube-api-access-wb2nm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:50.374305 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:50.374311    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa58022c-031e-45fb-9f00-d188fca88d48-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:54:51.035140 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:51.035104    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz" event={"ID":"fa58022c-031e-45fb-9f00-d188fca88d48","Type":"ContainerDied","Data":"0c5e40c3dd6cb708cc2b24d006bd06d6bda830ac30ef8145ee25f02c2937979a"}
Oct 31 12:54:51.035783 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:51.035748    5663 scope.go:122] "RemoveContainer" containerID="37743c0919a42cdddaad2fb2c29b75836a06c217630baaa001cef79ab77b1450"
Oct 31 12:54:51.036059 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:51.036045    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"
Oct 31 12:54:51.159298 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:51.159264    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"]
Oct 31 12:54:51.162291 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:51.162259    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b218d3-6cbf7cb6f4-8fwnz"]
Oct 31 12:54:51.464975 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:51.464940    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca29716-d6b0-4d20-8488-556d6018aefe" path="/var/lib/kubelet/pods/aca29716-d6b0-4d20-8488-556d6018aefe/volumes"
Oct 31 12:54:51.468310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:51.468279    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa58022c-031e-45fb-9f00-d188fca88d48" path="/var/lib/kubelet/pods/fa58022c-031e-45fb-9f00-d188fca88d48/volumes"
Oct 31 12:54:52.525375 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:54:52.525068    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:54:54.715157 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:54.715121    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"]
Oct 31 12:54:54.715691 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:54.715338    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy" containerID="cri-o://53c61f7118bb57ae0d445b4498bf657a6312f38a4aa3fe159400db630dac93d5" gracePeriod=30
Oct 31 12:54:55.176661 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.176625    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"]
Oct 31 12:54:55.177652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.177629    5663 cpu_manager.go:414] "containerMap: removing container" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy"
Oct 31 12:54:55.177757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.177655    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy"
Oct 31 12:54:55.177757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.177678    5663 cpu_manager.go:414] "containerMap: removing container" podUID="aca29716-d6b0-4d20-8488-556d6018aefe" containerName="payload-processing"
Oct 31 12:54:55.177757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.177687    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca29716-d6b0-4d20-8488-556d6018aefe" containerName="payload-processing"
Oct 31 12:54:55.177757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.177704    5663 cpu_manager.go:414] "containerMap: removing container" podUID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" containerName="istio-proxy"
Oct 31 12:54:55.177757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.177712    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" containerName="istio-proxy"
Oct 31 12:54:55.177757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.177734    5663 cpu_manager.go:414] "containerMap: removing container" podUID="fa58022c-031e-45fb-9f00-d188fca88d48" containerName="maas-api"
Oct 31 12:54:55.177757 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.177741    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa58022c-031e-45fb-9f00-d188fca88d48" containerName="maas-api"
Oct 31 12:54:55.177977 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.177773    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy"
Oct 31 12:54:55.177977 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.177780    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy"
Oct 31 12:54:55.178043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.178000    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c999a6-2915-4a7b-8f8f-2862bf0f880e" containerName="istio-proxy"
Oct 31 12:54:55.178043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.178026    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf969552-deff-4db0-91e7-45ff2b8a9c2b" containerName="istio-proxy"
Oct 31 12:54:55.178043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.178039    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca29716-d6b0-4d20-8488-556d6018aefe" containerName="payload-processing"
Oct 31 12:54:55.178185 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.178049    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="5855c115-9e7a-46b0-89e8-d7ec52b1719b" containerName="istio-proxy"
Oct 31 12:54:55.178185 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.178060    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa58022c-031e-45fb-9f00-d188fca88d48" containerName="maas-api"
Oct 31 12:54:55.189288 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.189264    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.197536 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.197495    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"]
Oct 31 12:54:55.214632 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.214609    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d176a5\"/\"kube-root-ca.crt\""
Oct 31 12:54:55.214809 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.214606    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d176a5\"/\"openshift-service-ca.crt\""
Oct 31 12:54:55.214940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.214825    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d176a5\"/\"default-dockercfg-7fl8n\""
Oct 31 12:54:55.215055 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.215030    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d176a5\"/\"test-model-d176a5-kserve-self-signed-certs\""
Oct 31 12:54:55.248935 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.248898    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-dshm\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.249088 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.249060    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-home\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.249164 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.249147    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kz6n\" (UniqueName: \"kubernetes.io/projected/3c7e399c-2f1e-4d61-8116-68b897e31ed1-kube-api-access-8kz6n\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.249219 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.249201    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tls-certs\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.249274 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.249240    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tmp-dir\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.249320 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.249275    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-model-cache\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.351431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.351395    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-dshm\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.351634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.351475    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-home\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.351634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.351522    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kz6n\" (UniqueName: \"kubernetes.io/projected/3c7e399c-2f1e-4d61-8116-68b897e31ed1-kube-api-access-8kz6n\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.351634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.351550    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tls-certs\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.351930 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.351892    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tmp-dir\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.351930 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.351918    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-home\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.352093 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.351960    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-model-cache\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.352201 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.352179    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tmp-dir\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.352406 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.352381    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-model-cache\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.353958 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.353930    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-dshm\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.354225 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.354205    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tls-certs\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.367230 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.367204    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kz6n\" (UniqueName: \"kubernetes.io/projected/3c7e399c-2f1e-4d61-8116-68b897e31ed1-kube-api-access-8kz6n\") pod \"test-model-d176a5-kserve-78cc77dcf9-pt4vv\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") " pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.517429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.517319    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:55.972258 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:55.972231    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"]
Oct 31 12:54:55.972764 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:54:55.972734    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c7e399c_2f1e_4d61_8116_68b897e31ed1.slice/crio-1f4074580cf4a044f1061ada8b5aef2ccd98d492602f14554b1436f9a35afa78 WatchSource:0}: Error finding container 1f4074580cf4a044f1061ada8b5aef2ccd98d492602f14554b1436f9a35afa78: Status 404 returned error can't find the container with id 1f4074580cf4a044f1061ada8b5aef2ccd98d492602f14554b1436f9a35afa78
Oct 31 12:54:56.086720 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:56.086674    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv" event={"ID":"3c7e399c-2f1e-4d61-8116-68b897e31ed1","Type":"ContainerStarted","Data":"bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043"}
Oct 31 12:54:56.086895 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:56.086733    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv" event={"ID":"3c7e399c-2f1e-4d61-8116-68b897e31ed1","Type":"ContainerStarted","Data":"1f4074580cf4a044f1061ada8b5aef2ccd98d492602f14554b1436f9a35afa78"}
Oct 31 12:54:56.519560 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:56.519520    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:54:58.209987 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.209919    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv" podStartSLOduration=3.209898297 podStartE2EDuration="3.209898297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:54:56.127016183 +0000 UTC m=+1620.867442400" watchObservedRunningTime="2026-10-31 12:54:58.209898297 +0000 UTC m=+1622.950324514"
Oct 31 12:54:58.212731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.212699    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"]
Oct 31 12:54:58.218321 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.218298    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.223528 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.223506    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2c393f\"/\"test-model-2c393f-kserve-self-signed-certs\""
Oct 31 12:54:58.223666 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.223595    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2c393f\"/\"openshift-service-ca.crt\""
Oct 31 12:54:58.224769 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.224743    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2c393f\"/\"default-dockercfg-qn9v8\""
Oct 31 12:54:58.225482 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.225461    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"]
Oct 31 12:54:58.287058 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.287011    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tmp-dir\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.287225 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.287135    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bszx\" (UniqueName: \"kubernetes.io/projected/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-kube-api-access-6bszx\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.287287 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.287258    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tls-certs\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.287435 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.287417    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-dshm\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.287575 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.287557    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-model-cache\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.287653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.287637    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-home\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.329915 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.329885    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2c393f\"/\"kube-root-ca.crt\""
Oct 31 12:54:58.389185 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.389147    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bszx\" (UniqueName: \"kubernetes.io/projected/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-kube-api-access-6bszx\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.389398 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.389216    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tls-certs\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.389398 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.389285    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-dshm\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.389398 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.389375    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-model-cache\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.389585 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.389433    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-home\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.389585 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.389520    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tmp-dir\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.389952 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.389919    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-model-cache\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.390061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.389984    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-home\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.390061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.390003    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tmp-dir\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.391836 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.391809    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-dshm\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.392226 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.392210    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tls-certs\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.403092 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.403064    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bszx\" (UniqueName: \"kubernetes.io/projected/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-kube-api-access-6bszx\") pod \"test-model-2c393f-kserve-548cc44567-q66p2\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") " pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:58.542435 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:58.539758    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:54:59.056396 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:54:59.056348    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5b51a9d_7551_43d7_85a1_8a61abd5cfc9.slice/crio-ca91400007be504c0317d037df49aaca72f1c6cdd4e5747cd51a97890c7e7ede WatchSource:0}: Error finding container ca91400007be504c0317d037df49aaca72f1c6cdd4e5747cd51a97890c7e7ede: Status 404 returned error can't find the container with id ca91400007be504c0317d037df49aaca72f1c6cdd4e5747cd51a97890c7e7ede
Oct 31 12:54:59.057307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.057283    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"]
Oct 31 12:54:59.123370 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.123327    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2" event={"ID":"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9","Type":"ContainerStarted","Data":"ca91400007be504c0317d037df49aaca72f1c6cdd4e5747cd51a97890c7e7ede"}
Oct 31 12:54:59.797426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.797388    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"]
Oct 31 12:54:59.802956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.802930    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.821422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.821366    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fd988c\"/\"kube-root-ca.crt\""
Oct 31 12:54:59.821422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.821420    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fd988c\"/\"openshift-service-ca.crt\""
Oct 31 12:54:59.821930 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.821903    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fd988c\"/\"test-model-fd988c-kserve-self-signed-certs\""
Oct 31 12:54:59.822226 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.822204    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fd988c\"/\"default-dockercfg-cbhjd\""
Oct 31 12:54:59.843257 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.843214    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-home\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.843652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.843559    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tls-certs\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.844074 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.844055    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brllg\" (UniqueName: \"kubernetes.io/projected/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-kube-api-access-brllg\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.844314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.844297    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-model-cache\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.844479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.844463    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tmp-dir\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.844629 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.844613    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-dshm\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.906332 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.904451    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"]
Oct 31 12:54:59.921469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.921438    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"]
Oct 31 12:54:59.921973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.921923    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy" containerID="cri-o://f599ed87ea81f98d02499ad7cff2ff4eb75587917bbe6d7bcdd815cd4fbf41cb" gracePeriod=30
Oct 31 12:54:59.924274 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.924249    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"]
Oct 31 12:54:59.924491 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.924455    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy" containerID="cri-o://8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f" gracePeriod=30
Oct 31 12:54:59.946019 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.945987    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-home\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.946329 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.946306    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tls-certs\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.946572 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.946552    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brllg\" (UniqueName: \"kubernetes.io/projected/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-kube-api-access-brllg\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.946727 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.946711    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-model-cache\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.946878 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.946863    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tmp-dir\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.947023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.947008    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-dshm\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.947237 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.947211    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-home\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.947509 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.947484    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-model-cache\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.948043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.948025    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tmp-dir\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.965945 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.965910    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-dshm\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.969338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.969281    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tls-certs\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:54:59.978889 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:54:59.978809    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brllg\" (UniqueName: \"kubernetes.io/projected/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-kube-api-access-brllg\") pod \"test-model-fd988c-kserve-6749b4d879-6wr64\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") " pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:55:00.129408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.128814    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:55:00.304430 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.300746    5663 generic.go:356] "Generic (PLEG): container finished" podID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerID="53c61f7118bb57ae0d445b4498bf657a6312f38a4aa3fe159400db630dac93d5" exitCode=0
Oct 31 12:55:00.304430 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.300887    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" event={"ID":"0a25fb34-1e28-435d-8ff9-430f2719f1f2","Type":"ContainerDied","Data":"53c61f7118bb57ae0d445b4498bf657a6312f38a4aa3fe159400db630dac93d5"}
Oct 31 12:55:00.315899 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.313681    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2" event={"ID":"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9","Type":"ContainerStarted","Data":"363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f"}
Oct 31 12:55:00.388935 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.385926    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2" podStartSLOduration=2.385906108 podStartE2EDuration="2.385906108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:00.338895533 +0000 UTC m=+1625.079321749" watchObservedRunningTime="2026-10-31 12:55:00.385906108 +0000 UTC m=+1625.126332325"
Oct 31 12:55:00.477427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.477380    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:55:00.495964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.495930    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-data\") pod \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") "
Oct 31 12:55:00.496171 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.495992    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-podinfo\") pod \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") "
Oct 31 12:55:00.496171 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.496025    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-credential-socket\") pod \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") "
Oct 31 12:55:00.496171 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.496068    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-envoy\") pod \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") "
Oct 31 12:55:00.496343 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.496171    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-certs\") pod \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") "
Oct 31 12:55:00.496343 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.496202    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-kube-api-access-mfbhm\" (UniqueName: \"kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-kube-api-access-mfbhm\") pod \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") "
Oct 31 12:55:00.496343 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.496292    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-token\" (UniqueName: \"kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-token\") pod \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") "
Oct 31 12:55:00.496508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.496346    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istiod-ca-cert\") pod \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") "
Oct 31 12:55:00.496677 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.496600    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-socket\") pod \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\" (UID: \"0a25fb34-1e28-435d-8ff9-430f2719f1f2\") "
Oct 31 12:55:00.498201 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.498176    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istiod-ca-cert" pod "0a25fb34-1e28-435d-8ff9-430f2719f1f2" (UID: "0a25fb34-1e28-435d-8ff9-430f2719f1f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:00.498459 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.498206    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-data" pod "0a25fb34-1e28-435d-8ff9-430f2719f1f2" (UID: "0a25fb34-1e28-435d-8ff9-430f2719f1f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:00.499147 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.499120    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-socket" pod "0a25fb34-1e28-435d-8ff9-430f2719f1f2" (UID: "0a25fb34-1e28-435d-8ff9-430f2719f1f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:00.499990 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.499963    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-certs" pod "0a25fb34-1e28-435d-8ff9-430f2719f1f2" (UID: "0a25fb34-1e28-435d-8ff9-430f2719f1f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:00.500329 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.500298    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-credential-socket" pod "0a25fb34-1e28-435d-8ff9-430f2719f1f2" (UID: "0a25fb34-1e28-435d-8ff9-430f2719f1f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:00.524577 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.520169    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-envoy" pod "0a25fb34-1e28-435d-8ff9-430f2719f1f2" (UID: "0a25fb34-1e28-435d-8ff9-430f2719f1f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:00.550260 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.547512    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-kube-api-access-mfbhm" pod "0a25fb34-1e28-435d-8ff9-430f2719f1f2" (UID: "0a25fb34-1e28-435d-8ff9-430f2719f1f2"). InnerVolumeSpecName "kube-api-access-mfbhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:00.550260 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.547610    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-podinfo" pod "0a25fb34-1e28-435d-8ff9-430f2719f1f2" (UID: "0a25fb34-1e28-435d-8ff9-430f2719f1f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:55:00.550260 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.550165    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:55:00.590486 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.562289    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-token" pod "0a25fb34-1e28-435d-8ff9-430f2719f1f2" (UID: "0a25fb34-1e28-435d-8ff9-430f2719f1f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:00.607268 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.602334    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.607268 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.602386    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.607268 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.602403    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.607268 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.602420    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.607268 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.602450    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.607268 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.602465    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a25fb34-1e28-435d-8ff9-430f2719f1f2-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.607268 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.602480    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfbhm\" (UniqueName: \"kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-kube-api-access-mfbhm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.607268 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.602495    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:00.607268 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:00.602509    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a25fb34-1e28-435d-8ff9-430f2719f1f2-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:01.097387 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.097340    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"]
Oct 31 12:55:01.098092 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:55:01.098063    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7343f75_22c7_4f7c_9e73_8edcdb5a18e4.slice/crio-7be7e111f054597e5872aa2e808acac0bcc07a50562cd84a13625e922b6cb9b6 WatchSource:0}: Error finding container 7be7e111f054597e5872aa2e808acac0bcc07a50562cd84a13625e922b6cb9b6: Status 404 returned error can't find the container with id 7be7e111f054597e5872aa2e808acac0bcc07a50562cd84a13625e922b6cb9b6
Oct 31 12:55:01.323444 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.323406    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64" event={"ID":"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4","Type":"ContainerStarted","Data":"88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45"}
Oct 31 12:55:01.323639 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.323454    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64" event={"ID":"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4","Type":"ContainerStarted","Data":"7be7e111f054597e5872aa2e808acac0bcc07a50562cd84a13625e922b6cb9b6"}
Oct 31 12:55:01.325118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.325094    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"
Oct 31 12:55:01.325247 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.325129    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5" event={"ID":"0a25fb34-1e28-435d-8ff9-430f2719f1f2","Type":"ContainerDied","Data":"6375b6df72f27b88c8eb02d88321b7ebb388f1a5119cba135aeccc0877963020"}
Oct 31 12:55:01.325247 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.325187    5663 scope.go:122] "RemoveContainer" containerID="53c61f7118bb57ae0d445b4498bf657a6312f38a4aa3fe159400db630dac93d5"
Oct 31 12:55:01.346837 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.346763    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64" podStartSLOduration=2.346749077 podStartE2EDuration="2.346749077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:01.343858961 +0000 UTC m=+1626.084285176" watchObservedRunningTime="2026-10-31 12:55:01.346749077 +0000 UTC m=+1626.087175328"
Oct 31 12:55:01.362535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.362505    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"]
Oct 31 12:55:01.365661 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.365640    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-dbbx5"]
Oct 31 12:55:01.443241 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.443199    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" path="/var/lib/kubelet/pods/0a25fb34-1e28-435d-8ff9-430f2719f1f2/volumes"
Oct 31 12:55:01.593023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.592985    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:55:01.593184 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.593049    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:55:01.644080 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.643997    5663 patch_prober.go:29] interesting pod/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:55:01.644080 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:01.644047    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:55:02.164095 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:02.164056    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:55:05.271266 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.271158    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.369499    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65535656-2606-4bda-b022-52ad8dd432b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65535656-2606-4bda-b022-52ad8dd432b2-istiod-ca-cert\") pod \"65535656-2606-4bda-b022-52ad8dd432b2\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") "
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.369559    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-istio-token\" (UniqueName: \"kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-istio-token\") pod \"65535656-2606-4bda-b022-52ad8dd432b2\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") "
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.369614    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-socket\") pod \"65535656-2606-4bda-b022-52ad8dd432b2\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") "
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.369737    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-certs\") pod \"65535656-2606-4bda-b022-52ad8dd432b2\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") "
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.369785    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65535656-2606-4bda-b022-52ad8dd432b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65535656-2606-4bda-b022-52ad8dd432b2-istio-podinfo\") pod \"65535656-2606-4bda-b022-52ad8dd432b2\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") "
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.369939    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-credential-socket\") pod \"65535656-2606-4bda-b022-52ad8dd432b2\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") "
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.369982    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-envoy\") pod \"65535656-2606-4bda-b022-52ad8dd432b2\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") "
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.370053    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-kube-api-access-69sq7\" (UniqueName: \"kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-kube-api-access-69sq7\") pod \"65535656-2606-4bda-b022-52ad8dd432b2\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") "
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.370104    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-data\") pod \"65535656-2606-4bda-b022-52ad8dd432b2\" (UID: \"65535656-2606-4bda-b022-52ad8dd432b2\") "
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.370291    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-socket" pod "65535656-2606-4bda-b022-52ad8dd432b2" (UID: "65535656-2606-4bda-b022-52ad8dd432b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.370657    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65535656-2606-4bda-b022-52ad8dd432b2-istiod-ca-cert" pod "65535656-2606-4bda-b022-52ad8dd432b2" (UID: "65535656-2606-4bda-b022-52ad8dd432b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.370898    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-data" pod "65535656-2606-4bda-b022-52ad8dd432b2" (UID: "65535656-2606-4bda-b022-52ad8dd432b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.371054    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.371074    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65535656-2606-4bda-b022-52ad8dd432b2-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.371090    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.371247    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-certs" pod "65535656-2606-4bda-b022-52ad8dd432b2" (UID: "65535656-2606-4bda-b022-52ad8dd432b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.372516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.371435    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-credential-socket" pod "65535656-2606-4bda-b022-52ad8dd432b2" (UID: "65535656-2606-4bda-b022-52ad8dd432b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.373870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.373818    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-envoy" pod "65535656-2606-4bda-b022-52ad8dd432b2" (UID: "65535656-2606-4bda-b022-52ad8dd432b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.374602 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.374572    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-kube-api-access-69sq7" pod "65535656-2606-4bda-b022-52ad8dd432b2" (UID: "65535656-2606-4bda-b022-52ad8dd432b2"). InnerVolumeSpecName "kube-api-access-69sq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:05.379792 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.379766    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-istio-token" pod "65535656-2606-4bda-b022-52ad8dd432b2" (UID: "65535656-2606-4bda-b022-52ad8dd432b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:05.379895 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.379862    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65535656-2606-4bda-b022-52ad8dd432b2-istio-podinfo" pod "65535656-2606-4bda-b022-52ad8dd432b2" (UID: "65535656-2606-4bda-b022-52ad8dd432b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:55:05.392388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.382473    5663 generic.go:356] "Generic (PLEG): container finished" podID="65535656-2606-4bda-b022-52ad8dd432b2" containerID="8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f" exitCode=0
Oct 31 12:55:05.392388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.382548    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" event={"ID":"65535656-2606-4bda-b022-52ad8dd432b2","Type":"ContainerDied","Data":"8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f"}
Oct 31 12:55:05.392388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.382613    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j" event={"ID":"65535656-2606-4bda-b022-52ad8dd432b2","Type":"ContainerDied","Data":"367182ec309822b936bc1b66443f6632b893fe8257c4bd66dabf39ba1e0a0702"}
Oct 31 12:55:05.392388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.382644    5663 scope.go:122] "RemoveContainer" containerID="8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f"
Oct 31 12:55:05.392388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.383945    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"
Oct 31 12:55:05.392388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.385396    5663 generic.go:356] "Generic (PLEG): container finished" podID="da085219-1b5e-4976-9f9b-45e98b491499" containerID="f599ed87ea81f98d02499ad7cff2ff4eb75587917bbe6d7bcdd815cd4fbf41cb" exitCode=0
Oct 31 12:55:05.392388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.385570    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" event={"ID":"da085219-1b5e-4976-9f9b-45e98b491499","Type":"ContainerDied","Data":"f599ed87ea81f98d02499ad7cff2ff4eb75587917bbe6d7bcdd815cd4fbf41cb"}
Oct 31 12:55:05.398430 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.398386    5663 scope.go:122] "RemoveContainer" containerID="8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f"
Oct 31 12:55:05.398745 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:55:05.398723    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f\": container with ID starting with 8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f not found: ID does not exist" containerID="8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f"
Oct 31 12:55:05.398802 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.398757    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f"} err="failed to get container status \"8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f\": rpc error: code = NotFound desc = could not find container \"8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f\": container with ID starting with 8b310f5290bb297bdf0b0dea9dc432edad21530bddcb423e9cc66419d1e5a31f not found: ID does not exist"
Oct 31 12:55:05.428645 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.428614    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"]
Oct 31 12:55:05.443236 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.443210    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-ccn6j"]
Oct 31 12:55:05.472604 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.472574    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65535656-2606-4bda-b022-52ad8dd432b2-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.472604 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.472603    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.472604 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.472618    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.472877 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.472631    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69sq7\" (UniqueName: \"kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-kube-api-access-69sq7\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.472877 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.472646    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65535656-2606-4bda-b022-52ad8dd432b2-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.472877 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.472658    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65535656-2606-4bda-b022-52ad8dd432b2-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.523890 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.519016    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:55:05.535614 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.535586    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:55:05.636741 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.636717    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:55:05.776466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.776378    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da085219-1b5e-4976-9f9b-45e98b491499-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da085219-1b5e-4976-9f9b-45e98b491499-istio-podinfo\") pod \"da085219-1b5e-4976-9f9b-45e98b491499\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") "
Oct 31 12:55:05.776630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.776481    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-certs\") pod \"da085219-1b5e-4976-9f9b-45e98b491499\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") "
Oct 31 12:55:05.776630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.776587    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-socket\") pod \"da085219-1b5e-4976-9f9b-45e98b491499\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") "
Oct 31 12:55:05.776630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.776616    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-envoy\") pod \"da085219-1b5e-4976-9f9b-45e98b491499\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") "
Oct 31 12:55:05.776804 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.776679    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-istio-token\" (UniqueName: \"kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-istio-token\") pod \"da085219-1b5e-4976-9f9b-45e98b491499\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") "
Oct 31 12:55:05.776804 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.776733    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da085219-1b5e-4976-9f9b-45e98b491499-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da085219-1b5e-4976-9f9b-45e98b491499-istiod-ca-cert\") pod \"da085219-1b5e-4976-9f9b-45e98b491499\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") "
Oct 31 12:55:05.776804 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.776770    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-data\") pod \"da085219-1b5e-4976-9f9b-45e98b491499\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") "
Oct 31 12:55:05.776804 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.776783    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-certs" pod "da085219-1b5e-4976-9f9b-45e98b491499" (UID: "da085219-1b5e-4976-9f9b-45e98b491499"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.777184 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.776865    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-socket" pod "da085219-1b5e-4976-9f9b-45e98b491499" (UID: "da085219-1b5e-4976-9f9b-45e98b491499"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.777768 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.777486    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da085219-1b5e-4976-9f9b-45e98b491499-istiod-ca-cert" pod "da085219-1b5e-4976-9f9b-45e98b491499" (UID: "da085219-1b5e-4976-9f9b-45e98b491499"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:05.778335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.778310    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-kube-api-access-jcxsb\" (UniqueName: \"kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-kube-api-access-jcxsb\") pod \"da085219-1b5e-4976-9f9b-45e98b491499\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") "
Oct 31 12:55:05.778477 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.778391    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-credential-socket\") pod \"da085219-1b5e-4976-9f9b-45e98b491499\" (UID: \"da085219-1b5e-4976-9f9b-45e98b491499\") "
Oct 31 12:55:05.779114 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.779093    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da085219-1b5e-4976-9f9b-45e98b491499-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.779239 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.779120    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.779239 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.779134    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.780443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.779498    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da085219-1b5e-4976-9f9b-45e98b491499-istio-podinfo" pod "da085219-1b5e-4976-9f9b-45e98b491499" (UID: "da085219-1b5e-4976-9f9b-45e98b491499"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:55:05.780443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.779527    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-istio-token" pod "da085219-1b5e-4976-9f9b-45e98b491499" (UID: "da085219-1b5e-4976-9f9b-45e98b491499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:05.780443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.779528    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-credential-socket" pod "da085219-1b5e-4976-9f9b-45e98b491499" (UID: "da085219-1b5e-4976-9f9b-45e98b491499"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.780443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.779557    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-data" pod "da085219-1b5e-4976-9f9b-45e98b491499" (UID: "da085219-1b5e-4976-9f9b-45e98b491499"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.780443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.779568    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-envoy" pod "da085219-1b5e-4976-9f9b-45e98b491499" (UID: "da085219-1b5e-4976-9f9b-45e98b491499"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:05.780975 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.780948    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-kube-api-access-jcxsb" pod "da085219-1b5e-4976-9f9b-45e98b491499" (UID: "da085219-1b5e-4976-9f9b-45e98b491499"). InnerVolumeSpecName "kube-api-access-jcxsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:05.881521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.881488    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da085219-1b5e-4976-9f9b-45e98b491499-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.881521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.881518    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.881521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.881528    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.881748 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.881536    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.881748 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.881546    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcxsb\" (UniqueName: \"kubernetes.io/projected/da085219-1b5e-4976-9f9b-45e98b491499-kube-api-access-jcxsb\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:05.881748 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:05.881554    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da085219-1b5e-4976-9f9b-45e98b491499-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:06.048471 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:06.048376    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"]
Oct 31 12:55:06.076147 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:06.076117    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"]
Oct 31 12:55:06.393850 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:06.393816    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"
Oct 31 12:55:06.393850 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:06.393836    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc" event={"ID":"da085219-1b5e-4976-9f9b-45e98b491499","Type":"ContainerDied","Data":"0ff7c74d556969e155c3bd43b8279f309564f305938afeea21033b6fe6931b93"}
Oct 31 12:55:06.394422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:06.393924    5663 scope.go:122] "RemoveContainer" containerID="f599ed87ea81f98d02499ad7cff2ff4eb75587917bbe6d7bcdd815cd4fbf41cb"
Oct 31 12:55:06.409450 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:06.409424    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:55:06.426052 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:06.426020    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"]
Oct 31 12:55:06.429094 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:06.429072    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b218d3-openshift-default-67596dd89f-rxzcc"]
Oct 31 12:55:07.441216 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:07.441179    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65535656-2606-4bda-b022-52ad8dd432b2" path="/var/lib/kubelet/pods/65535656-2606-4bda-b022-52ad8dd432b2/volumes"
Oct 31 12:55:07.441753 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:07.441737    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da085219-1b5e-4976-9f9b-45e98b491499" path="/var/lib/kubelet/pods/da085219-1b5e-4976-9f9b-45e98b491499/volumes"
Oct 31 12:55:08.578673 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:08.578635    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:55:08.612088 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:08.612057    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:55:09.428446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:09.428417    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:55:10.164536 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:10.164503    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:55:10.177819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:10.177787    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:55:10.533182 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:10.533103    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:55:11.431037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:11.431003    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"]
Oct 31 12:55:16.431974 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:16.431918    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8" podUID="f06c9a3d-a297-4211-aa4d-73ab00ec9aa0" containerName="payload-pre-processing" containerID="cri-o://96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e" gracePeriod=25
Oct 31 12:55:19.776758 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:19.776718    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"]
Oct 31 12:55:21.080992 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.080925    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main" containerID="cri-o://e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5" gracePeriod=45
Oct 31 12:55:21.102388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.102315    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" containerID="cri-o://947c09b8acf76a64abb424d296a32a29eb3351fe41a87bb537162944811522d0" gracePeriod=45
Oct 31 12:55:21.797217 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.796780    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:55:21.800552 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.800460    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:55:21.833163 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.833127    5663 generic.go:356] "Generic (PLEG): container finished" podID="118349cf-bd83-48fb-860a-42ec40ab2285" containerID="e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5" exitCode=0
Oct 31 12:55:21.833635 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.833605    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" event={"ID":"118349cf-bd83-48fb-860a-42ec40ab2285","Type":"ContainerDied","Data":"e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5"}
Oct 31 12:55:21.833823 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.833806    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl" event={"ID":"118349cf-bd83-48fb-860a-42ec40ab2285","Type":"ContainerDied","Data":"db374a13aab2302b32d5c9abcdb0afb50bb36c76e5cf749d0425ee1961d312d2"}
Oct 31 12:55:21.833934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.833922    5663 scope.go:122] "RemoveContainer" containerID="e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5"
Oct 31 12:55:21.834212 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.834194    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"
Oct 31 12:55:21.879966 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.879327    5663 generic.go:356] "Generic (PLEG): container finished" podID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerID="947c09b8acf76a64abb424d296a32a29eb3351fe41a87bb537162944811522d0" exitCode=0
Oct 31 12:55:21.879966 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.879422    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" event={"ID":"2cb2a3c0-7255-4be3-93c4-55d9ee673aad","Type":"ContainerDied","Data":"947c09b8acf76a64abb424d296a32a29eb3351fe41a87bb537162944811522d0"}
Oct 31 12:55:21.879966 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.879509    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"
Oct 31 12:55:21.919927 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.919258    5663 scope.go:122] "RemoveContainer" containerID="e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5"
Oct 31 12:55:21.920216 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:55:21.920181    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5\": container with ID starting with e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5 not found: ID does not exist" containerID="e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5"
Oct 31 12:55:21.920320 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.920228    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5"} err="failed to get container status \"e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5\": rpc error: code = NotFound desc = could not find container \"e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5\": container with ID starting with e4b8b20965d0fd5f74903e05251d670d04ac4e431a6e1bc98082dd507ebcc4b5 not found: ID does not exist"
Oct 31 12:55:21.920320 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.920253    5663 scope.go:122] "RemoveContainer" containerID="947c09b8acf76a64abb424d296a32a29eb3351fe41a87bb537162944811522d0"
Oct 31 12:55:21.998547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.998507    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/118349cf-bd83-48fb-860a-42ec40ab2285-kube-api-access-wvrmd\" (UniqueName: \"kubernetes.io/projected/118349cf-bd83-48fb-860a-42ec40ab2285-kube-api-access-wvrmd\") pod \"118349cf-bd83-48fb-860a-42ec40ab2285\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") "
Oct 31 12:55:21.998744 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.998590    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-dshm\") pod \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") "
Oct 31 12:55:21.998744 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.998645    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-home\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-home\") pod \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") "
Oct 31 12:55:21.998744 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.998732    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-model-cache\") pod \"118349cf-bd83-48fb-860a-42ec40ab2285\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") "
Oct 31 12:55:21.998917 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.998778    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/118349cf-bd83-48fb-860a-42ec40ab2285-tls-certs\" (UniqueName: \"kubernetes.io/secret/118349cf-bd83-48fb-860a-42ec40ab2285-tls-certs\") pod \"118349cf-bd83-48fb-860a-42ec40ab2285\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") "
Oct 31 12:55:21.998917 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.998810    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-kube-api-access-nzlnn\" (UniqueName: \"kubernetes.io/projected/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-kube-api-access-nzlnn\") pod \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") "
Oct 31 12:55:21.998917 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.998872    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-dshm\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-dshm\") pod \"118349cf-bd83-48fb-860a-42ec40ab2285\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") "
Oct 31 12:55:21.999072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.998914    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-tmp-dir\") pod \"118349cf-bd83-48fb-860a-42ec40ab2285\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") "
Oct 31 12:55:21.999072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.998956    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-home" pod "2cb2a3c0-7255-4be3-93c4-55d9ee673aad" (UID: "2cb2a3c0-7255-4be3-93c4-55d9ee673aad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:21.999072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.998958    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-model-cache" pod "118349cf-bd83-48fb-860a-42ec40ab2285" (UID: "118349cf-bd83-48fb-860a-42ec40ab2285"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:21.999072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.998964    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-home\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-home\") pod \"118349cf-bd83-48fb-860a-42ec40ab2285\" (UID: \"118349cf-bd83-48fb-860a-42ec40ab2285\") "
Oct 31 12:55:21.999278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.999090    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-model-cache\") pod \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") "
Oct 31 12:55:21.999278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.999170    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-home" pod "118349cf-bd83-48fb-860a-42ec40ab2285" (UID: "118349cf-bd83-48fb-860a-42ec40ab2285"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:21.999278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.999182    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-tmp-dir" pod "118349cf-bd83-48fb-860a-42ec40ab2285" (UID: "118349cf-bd83-48fb-860a-42ec40ab2285"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:21.999278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.999261    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tls-certs\") pod \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") "
Oct 31 12:55:21.999496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.999316    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tmp-dir\") pod \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\" (UID: \"2cb2a3c0-7255-4be3-93c4-55d9ee673aad\") "
Oct 31 12:55:21.999496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.999316    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-model-cache" pod "2cb2a3c0-7255-4be3-93c4-55d9ee673aad" (UID: "2cb2a3c0-7255-4be3-93c4-55d9ee673aad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:21.999607 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.999546    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tmp-dir" pod "2cb2a3c0-7255-4be3-93c4-55d9ee673aad" (UID: "2cb2a3c0-7255-4be3-93c4-55d9ee673aad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:21.999973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.999953    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:21.999973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.999972    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:22.000134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.999982    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:22.000134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.999991    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:22.000134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:21.999999    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:22.000134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.000008    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:22.001530 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.001504    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-dshm" pod "118349cf-bd83-48fb-860a-42ec40ab2285" (UID: "118349cf-bd83-48fb-860a-42ec40ab2285"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:22.001655 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.001555    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118349cf-bd83-48fb-860a-42ec40ab2285-kube-api-access-wvrmd" pod "118349cf-bd83-48fb-860a-42ec40ab2285" (UID: "118349cf-bd83-48fb-860a-42ec40ab2285"). InnerVolumeSpecName "kube-api-access-wvrmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:22.001655 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.001609    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-kube-api-access-nzlnn" pod "2cb2a3c0-7255-4be3-93c4-55d9ee673aad" (UID: "2cb2a3c0-7255-4be3-93c4-55d9ee673aad"). InnerVolumeSpecName "kube-api-access-nzlnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:22.001655 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.001618    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-dshm" pod "2cb2a3c0-7255-4be3-93c4-55d9ee673aad" (UID: "2cb2a3c0-7255-4be3-93c4-55d9ee673aad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:22.001655 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.001615    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tls-certs" pod "2cb2a3c0-7255-4be3-93c4-55d9ee673aad" (UID: "2cb2a3c0-7255-4be3-93c4-55d9ee673aad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:55:22.002212 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.002197    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118349cf-bd83-48fb-860a-42ec40ab2285-tls-certs" pod "118349cf-bd83-48fb-860a-42ec40ab2285" (UID: "118349cf-bd83-48fb-860a-42ec40ab2285"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:55:22.101158 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.101122    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118349cf-bd83-48fb-860a-42ec40ab2285-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:22.101158 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.101155    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzlnn\" (UniqueName: \"kubernetes.io/projected/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-kube-api-access-nzlnn\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:22.101158 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.101166    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/118349cf-bd83-48fb-860a-42ec40ab2285-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:22.101736 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.101174    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:22.101736 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.101183    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvrmd\" (UniqueName: \"kubernetes.io/projected/118349cf-bd83-48fb-860a-42ec40ab2285-kube-api-access-wvrmd\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:22.101736 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.101191    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a3c0-7255-4be3-93c4-55d9ee673aad-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:22.180438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.180348    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"]
Oct 31 12:55:22.184177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.184149    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/distinct-w2-e6c79d-kserve-86bc6fcf99-v6qxl"]
Oct 31 12:55:22.215887 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.215853    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"]
Oct 31 12:55:22.229454 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.229426    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq"]
Oct 31 12:55:22.477239 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:55:22.477160    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:55:22.779195 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:22.779103    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e6c79d/simulator-w2-e6c79d-kserve-76c67955d5-nvkbq" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 31 12:55:23.440187 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:23.440148    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" path="/var/lib/kubelet/pods/118349cf-bd83-48fb-860a-42ec40ab2285/volumes"
Oct 31 12:55:23.440681 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:23.440666    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" path="/var/lib/kubelet/pods/2cb2a3c0-7255-4be3-93c4-55d9ee673aad/volumes"
Oct 31 12:55:25.800707 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:25.793410    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8" podUID="f06c9a3d-a297-4211-aa4d-73ab00ec9aa0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:55:31.599669 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:31.599630    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"]
Oct 31 12:55:34.811649 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:34.811612    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4" podUID="2b7f4674-4856-4c0a-820c-24a59546c885" containerName="main" containerID="cri-o://44aad1c4b157417bbad9a984ccca570d7863354614a3a85163c4c984f2e8a8b6" gracePeriod=45
Oct 31 12:55:34.944955 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:34.944910    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4" podUID="2b7f4674-4856-4c0a-820c-24a59546c885" containerName="main" probeResult="failure" output="Get \"https://10.133.0.121:8000/ready\": dial tcp 10.133.0.121:8000: connect: connection refused"
Oct 31 12:55:35.031699 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.031650    5663 generic.go:356] "Generic (PLEG): container finished" podID="2b7f4674-4856-4c0a-820c-24a59546c885" containerID="44aad1c4b157417bbad9a984ccca570d7863354614a3a85163c4c984f2e8a8b6" exitCode=0
Oct 31 12:55:35.031887 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.031734    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4" event={"ID":"2b7f4674-4856-4c0a-820c-24a59546c885","Type":"ContainerDied","Data":"44aad1c4b157417bbad9a984ccca570d7863354614a3a85163c4c984f2e8a8b6"}
Oct 31 12:55:35.101405 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.101381    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:55:35.277768 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.277730    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-tmp-dir\") pod \"2b7f4674-4856-4c0a-820c-24a59546c885\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") "
Oct 31 12:55:35.277947 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.277861    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-home\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-home\") pod \"2b7f4674-4856-4c0a-820c-24a59546c885\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") "
Oct 31 12:55:35.277947 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.277921    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-model-cache\") pod \"2b7f4674-4856-4c0a-820c-24a59546c885\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") "
Oct 31 12:55:35.278066 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.277960    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-tmp-dir" pod "2b7f4674-4856-4c0a-820c-24a59546c885" (UID: "2b7f4674-4856-4c0a-820c-24a59546c885"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:35.278066 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.278046    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-dshm\") pod \"2b7f4674-4856-4c0a-820c-24a59546c885\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") "
Oct 31 12:55:35.278177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.278089    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7f4674-4856-4c0a-820c-24a59546c885-kube-api-access-zc6sk\" (UniqueName: \"kubernetes.io/projected/2b7f4674-4856-4c0a-820c-24a59546c885-kube-api-access-zc6sk\") pod \"2b7f4674-4856-4c0a-820c-24a59546c885\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") "
Oct 31 12:55:35.278238 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.278184    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b7f4674-4856-4c0a-820c-24a59546c885-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7f4674-4856-4c0a-820c-24a59546c885-tls-certs\") pod \"2b7f4674-4856-4c0a-820c-24a59546c885\" (UID: \"2b7f4674-4856-4c0a-820c-24a59546c885\") "
Oct 31 12:55:35.278238 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.278165    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-home" pod "2b7f4674-4856-4c0a-820c-24a59546c885" (UID: "2b7f4674-4856-4c0a-820c-24a59546c885"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:35.278238 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.278216    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-model-cache" pod "2b7f4674-4856-4c0a-820c-24a59546c885" (UID: "2b7f4674-4856-4c0a-820c-24a59546c885"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:35.278803 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.278770    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:35.278803 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.278796    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:35.279003 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.278811    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:35.280392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.280348    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7f4674-4856-4c0a-820c-24a59546c885-tls-certs" pod "2b7f4674-4856-4c0a-820c-24a59546c885" (UID: "2b7f4674-4856-4c0a-820c-24a59546c885"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:55:35.280392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.280343    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-dshm" pod "2b7f4674-4856-4c0a-820c-24a59546c885" (UID: "2b7f4674-4856-4c0a-820c-24a59546c885"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:35.280551 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.280468    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7f4674-4856-4c0a-820c-24a59546c885-kube-api-access-zc6sk" pod "2b7f4674-4856-4c0a-820c-24a59546c885" (UID: "2b7f4674-4856-4c0a-820c-24a59546c885"). InnerVolumeSpecName "kube-api-access-zc6sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:35.380814 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.380732    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7f4674-4856-4c0a-820c-24a59546c885-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:35.380814 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.380765    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc6sk\" (UniqueName: \"kubernetes.io/projected/2b7f4674-4856-4c0a-820c-24a59546c885-kube-api-access-zc6sk\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:35.380814 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.380778    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7f4674-4856-4c0a-820c-24a59546c885-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:35.762463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:35.762332    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8" podUID="f06c9a3d-a297-4211-aa4d-73ab00ec9aa0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:55:36.040233 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:36.040158    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"
Oct 31 12:55:36.040233 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:36.040187    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4" event={"ID":"2b7f4674-4856-4c0a-820c-24a59546c885","Type":"ContainerDied","Data":"0c86cce6445ce46000c9b948459d395f0ff34e8eb5fd207392433ff8d39e5ef3"}
Oct 31 12:55:36.040760 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:36.040244    5663 scope.go:122] "RemoveContainer" containerID="44aad1c4b157417bbad9a984ccca570d7863354614a3a85163c4c984f2e8a8b6"
Oct 31 12:55:36.061160 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:36.061134    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"]
Oct 31 12:55:36.065314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:36.065289    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-015701/test-model-015701-kserve-6b595fcd55-b4ff4"]
Oct 31 12:55:36.711665 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:36.711598    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"
Oct 31 12:55:36.806887 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:36.806853    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-kube-api-access-tkdq6\" (UniqueName: \"kubernetes.io/projected/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-kube-api-access-tkdq6\") pod \"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0\" (UID: \"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0\") "
Oct 31 12:55:36.807059 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:36.806997    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-plugins-config-volume\") pod \"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0\" (UID: \"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0\") "
Oct 31 12:55:36.807404 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:36.807377    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-plugins-config-volume" pod "f06c9a3d-a297-4211-aa4d-73ab00ec9aa0" (UID: "f06c9a3d-a297-4211-aa4d-73ab00ec9aa0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:36.807665 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:36.807645    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:36.809015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:36.808992    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-kube-api-access-tkdq6" pod "f06c9a3d-a297-4211-aa4d-73ab00ec9aa0" (UID: "f06c9a3d-a297-4211-aa4d-73ab00ec9aa0"). InnerVolumeSpecName "kube-api-access-tkdq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:36.909196 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:36.909152    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkdq6\" (UniqueName: \"kubernetes.io/projected/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0-kube-api-access-tkdq6\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:37.049488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.049397    5663 generic.go:356] "Generic (PLEG): container finished" podID="f06c9a3d-a297-4211-aa4d-73ab00ec9aa0" containerID="96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e" exitCode=0
Oct 31 12:55:37.049488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.049453    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"
Oct 31 12:55:37.049488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.049473    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8" event={"ID":"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0","Type":"ContainerDied","Data":"96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e"}
Oct 31 12:55:37.050043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.049569    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8" event={"ID":"f06c9a3d-a297-4211-aa4d-73ab00ec9aa0","Type":"ContainerDied","Data":"ac53ecb01a604ff047bbdf4bcbd93e12caa80a312121eb239279a88b8af38068"}
Oct 31 12:55:37.050043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.049603    5663 scope.go:122] "RemoveContainer" containerID="96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e"
Oct 31 12:55:37.062654 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.062630    5663 scope.go:122] "RemoveContainer" containerID="96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e"
Oct 31 12:55:37.062976 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:55:37.062955    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e\": container with ID starting with 96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e not found: ID does not exist" containerID="96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e"
Oct 31 12:55:37.063051 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.062984    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e"} err="failed to get container status \"96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e\": rpc error: code = NotFound desc = could not find container \"96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e\": container with ID starting with 96c78bff5f54c6ebfff47621faa091a298fd049bc60085add1870a08dd379d0e not found: ID does not exist"
Oct 31 12:55:37.074269 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.074247    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"]
Oct 31 12:55:37.078557 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.078533    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e6c79d-86d79b4c67-4l7l8"]
Oct 31 12:55:37.191329 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.191283    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"]
Oct 31 12:55:37.198697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198662    5663 cpu_manager.go:414] "containerMap: removing container" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy"
Oct 31 12:55:37.198697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198696    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy"
Oct 31 12:55:37.198940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198744    5663 cpu_manager.go:414] "containerMap: removing container" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main"
Oct 31 12:55:37.198940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198753    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main"
Oct 31 12:55:37.198940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198768    5663 cpu_manager.go:414] "containerMap: removing container" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy"
Oct 31 12:55:37.198940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198776    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy"
Oct 31 12:55:37.198940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198797    5663 cpu_manager.go:414] "containerMap: removing container" podUID="2b7f4674-4856-4c0a-820c-24a59546c885" containerName="main"
Oct 31 12:55:37.198940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198804    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7f4674-4856-4c0a-820c-24a59546c885" containerName="main"
Oct 31 12:55:37.198940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198818    5663 cpu_manager.go:414] "containerMap: removing container" podUID="f06c9a3d-a297-4211-aa4d-73ab00ec9aa0" containerName="payload-pre-processing"
Oct 31 12:55:37.198940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198827    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06c9a3d-a297-4211-aa4d-73ab00ec9aa0" containerName="payload-pre-processing"
Oct 31 12:55:37.198940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198841    5663 cpu_manager.go:414] "containerMap: removing container" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main"
Oct 31 12:55:37.198940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198848    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main"
Oct 31 12:55:37.198940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198877    5663 cpu_manager.go:414] "containerMap: removing container" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy"
Oct 31 12:55:37.198940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.198885    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy"
Oct 31 12:55:37.199546 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.199036    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a25fb34-1e28-435d-8ff9-430f2719f1f2" containerName="istio-proxy"
Oct 31 12:55:37.199546 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.199055    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="118349cf-bd83-48fb-860a-42ec40ab2285" containerName="main"
Oct 31 12:55:37.199546 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.199064    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="65535656-2606-4bda-b022-52ad8dd432b2" containerName="istio-proxy"
Oct 31 12:55:37.199546 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.199076    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7f4674-4856-4c0a-820c-24a59546c885" containerName="main"
Oct 31 12:55:37.199546 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.199086    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06c9a3d-a297-4211-aa4d-73ab00ec9aa0" containerName="payload-pre-processing"
Oct 31 12:55:37.199546 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.199096    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb2a3c0-7255-4be3-93c4-55d9ee673aad" containerName="main"
Oct 31 12:55:37.199546 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.199109    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="da085219-1b5e-4976-9f9b-45e98b491499" containerName="istio-proxy"
Oct 31 12:55:37.205980 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.205901    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.214422 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.214395    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.214745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.214723    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.214889 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.214870    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.215101 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.215081    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrxc\" (UniqueName: \"kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-kube-api-access-qbrxc\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.215307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.215291    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.215477 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.215459    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.215647 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.215630    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.215762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.215748    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.215865 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.215851    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.224911 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.224851    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"]
Oct 31 12:55:37.236763 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.236702    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d086f3-openshift-default-dockercfg-hcwf6\""
Oct 31 12:55:37.317598 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.317054    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.317598 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.317128    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.317598 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.317160    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.317598 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.317237    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.317598 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.317321    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.317598 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.317347    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.317598 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.317452    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrxc\" (UniqueName: \"kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-kube-api-access-qbrxc\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.317598 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.317556    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.317598 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.317600    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.318521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.318257    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.318521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.318434    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.318652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.318635    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.318744 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.318718    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.318812 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.318748    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.320875 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.320849    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.321023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.320999    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.330627 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.330602    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.330770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.330753    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrxc\" (UniqueName: \"kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-kube-api-access-qbrxc\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.441471 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.441437    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b7f4674-4856-4c0a-820c-24a59546c885" path="/var/lib/kubelet/pods/2b7f4674-4856-4c0a-820c-24a59546c885/volumes"
Oct 31 12:55:37.441941 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.441923    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06c9a3d-a297-4211-aa4d-73ab00ec9aa0" path="/var/lib/kubelet/pods/f06c9a3d-a297-4211-aa4d-73ab00ec9aa0/volumes"
Oct 31 12:55:37.526961 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.526925    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:37.819082 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.819051    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"]
Oct 31 12:55:37.820855 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:55:37.820820    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae245313_05e0_4c03_aa7f_3d2e9eabf675.slice/crio-24417806c25ab1343f3923b31f1394ffe4da91c5e3f3630e8ac454382c8154d5 WatchSource:0}: Error finding container 24417806c25ab1343f3923b31f1394ffe4da91c5e3f3630e8ac454382c8154d5: Status 404 returned error can't find the container with id 24417806c25ab1343f3923b31f1394ffe4da91c5e3f3630e8ac454382c8154d5
Oct 31 12:55:37.823682 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.823640    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:37.823751 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.823733    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:37.823794 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:37.823783    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:55:38.118223 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:38.116994    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" event={"ID":"ae245313-05e0-4c03-aa7f-3d2e9eabf675","Type":"ContainerStarted","Data":"59451011cbe7844a074c3d17f986c7b985ba344643b6d2a5225f95142dfb2c83"}
Oct 31 12:55:38.118223 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:38.117063    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" event={"ID":"ae245313-05e0-4c03-aa7f-3d2e9eabf675","Type":"ContainerStarted","Data":"24417806c25ab1343f3923b31f1394ffe4da91c5e3f3630e8ac454382c8154d5"}
Oct 31 12:55:38.148527 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:38.147646    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" podStartSLOduration=1.147631868 podStartE2EDuration="1.147631868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:55:38.144000575 +0000 UTC m=+1662.884426791" watchObservedRunningTime="2026-10-31 12:55:38.147631868 +0000 UTC m=+1662.888058084"
Oct 31 12:55:38.532028 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:38.531947    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:38.532418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:38.532391    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:55:38.532624 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:38.532593    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 12:55:39.532376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:39.529877    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:55:39.532376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:39.529939    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 12:55:40.527786 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:40.527753    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:55:40.527960 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:40.527801    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 12:55:41.527429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:41.527393    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:55:41.527813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:41.527447    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 12:55:42.379438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:42.379401    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"]
Oct 31 12:55:42.405436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:42.405406    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"]
Oct 31 12:55:42.405927 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:42.405872    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" podUID="9780b010-8c66-4490-8f1a-82296b85235e" containerName="istio-proxy" containerID="cri-o://0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83" gracePeriod=30
Oct 31 12:55:42.441450 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:42.441408    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"]
Oct 31 12:55:42.441901 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:42.441869    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" podUID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" containerName="istio-proxy" containerID="cri-o://0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394" gracePeriod=30
Oct 31 12:55:42.552261 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:42.546002    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:55:42.552261 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:42.546077    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 12:55:43.530325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:43.529474    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:55:43.530325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:43.529535    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 12:55:44.543395 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:44.543010    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 12:55:44.543395 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:44.543070    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 12:55:45.658613 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:45.658579    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:45.660670 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:45.660645    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:45.662466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:45.662442    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:55:46.558793 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:46.558756    5663 patch_prober.go:29] interesting pod/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:55:46.558952 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:46.558816    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" podUID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:55:46.640838 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:46.640765    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn" podUID="1d224a67-a771-46d2-ad68-c8970ca2dffa" containerName="main" containerID="cri-o://00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab" gracePeriod=45
Oct 31 12:55:46.915640 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:46.915613    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:55:47.021466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.021339    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-home" pod "1d224a67-a771-46d2-ad68-c8970ca2dffa" (UID: "1d224a67-a771-46d2-ad68-c8970ca2dffa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.022981 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.018311    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-home\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-home\") pod \"1d224a67-a771-46d2-ad68-c8970ca2dffa\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") "
Oct 31 12:55:47.023248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.023232    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-dshm\") pod \"1d224a67-a771-46d2-ad68-c8970ca2dffa\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") "
Oct 31 12:55:47.023873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.023855    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d224a67-a771-46d2-ad68-c8970ca2dffa-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d224a67-a771-46d2-ad68-c8970ca2dffa-tls-certs\") pod \"1d224a67-a771-46d2-ad68-c8970ca2dffa\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") "
Oct 31 12:55:47.024411 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.024392    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-model-cache\") pod \"1d224a67-a771-46d2-ad68-c8970ca2dffa\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") "
Oct 31 12:55:47.024820 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.024804    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d224a67-a771-46d2-ad68-c8970ca2dffa-kube-api-access-5p9sp\" (UniqueName: \"kubernetes.io/projected/1d224a67-a771-46d2-ad68-c8970ca2dffa-kube-api-access-5p9sp\") pod \"1d224a67-a771-46d2-ad68-c8970ca2dffa\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") "
Oct 31 12:55:47.025842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.024735    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-model-cache" pod "1d224a67-a771-46d2-ad68-c8970ca2dffa" (UID: "1d224a67-a771-46d2-ad68-c8970ca2dffa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.026041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.026022    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-tmp-dir\") pod \"1d224a67-a771-46d2-ad68-c8970ca2dffa\" (UID: \"1d224a67-a771-46d2-ad68-c8970ca2dffa\") "
Oct 31 12:55:47.033597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.026333    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-tmp-dir" pod "1d224a67-a771-46d2-ad68-c8970ca2dffa" (UID: "1d224a67-a771-46d2-ad68-c8970ca2dffa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.033597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.027777    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.033597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.027802    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.033597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.027817    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.035236 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.034014    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d224a67-a771-46d2-ad68-c8970ca2dffa-kube-api-access-5p9sp" pod "1d224a67-a771-46d2-ad68-c8970ca2dffa" (UID: "1d224a67-a771-46d2-ad68-c8970ca2dffa"). InnerVolumeSpecName "kube-api-access-5p9sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:47.043419 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.043344    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-dshm" pod "1d224a67-a771-46d2-ad68-c8970ca2dffa" (UID: "1d224a67-a771-46d2-ad68-c8970ca2dffa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.043679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.043650    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d224a67-a771-46d2-ad68-c8970ca2dffa-tls-certs" pod "1d224a67-a771-46d2-ad68-c8970ca2dffa" (UID: "1d224a67-a771-46d2-ad68-c8970ca2dffa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:55:47.129084 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.129006    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d224a67-a771-46d2-ad68-c8970ca2dffa-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.129084 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.129034    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p9sp\" (UniqueName: \"kubernetes.io/projected/1d224a67-a771-46d2-ad68-c8970ca2dffa-kube-api-access-5p9sp\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.129084 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.129044    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d224a67-a771-46d2-ad68-c8970ca2dffa-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.278508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.278468    5663 generic.go:356] "Generic (PLEG): container finished" podID="1d224a67-a771-46d2-ad68-c8970ca2dffa" containerID="00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab" exitCode=0
Oct 31 12:55:47.278700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.278518    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"
Oct 31 12:55:47.278700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.278552    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn" event={"ID":"1d224a67-a771-46d2-ad68-c8970ca2dffa","Type":"ContainerDied","Data":"00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab"}
Oct 31 12:55:47.278700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.278603    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn" event={"ID":"1d224a67-a771-46d2-ad68-c8970ca2dffa","Type":"ContainerDied","Data":"9c033b6ae3eb7ac2a61fd5ad6e15f0e008c546faa0ed9b00fb0255ed61ef71ef"}
Oct 31 12:55:47.278700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.278626    5663 scope.go:122] "RemoveContainer" containerID="00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab"
Oct 31 12:55:47.289015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.288999    5663 scope.go:122] "RemoveContainer" containerID="00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab"
Oct 31 12:55:47.289257 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:55:47.289235    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab\": container with ID starting with 00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab not found: ID does not exist" containerID="00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab"
Oct 31 12:55:47.289325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.289267    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab"} err="failed to get container status \"00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab\": rpc error: code = NotFound desc = could not find container \"00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab\": container with ID starting with 00db64ebd6a5acded31f85c1ac009c0ea4734e913afc517a415136d52228ddab not found: ID does not exist"
Oct 31 12:55:47.361116 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.361082    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"]
Oct 31 12:55:47.366374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.366331    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ad1af7/test-model-ad1af7-kserve-b4db66dc9-b99sn"]
Oct 31 12:55:47.380884 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.380806    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx" podUID="3dc43894-bca9-4020-a8a6-923731346094" containerName="payload-processing" containerID="cri-o://fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807" gracePeriod=25
Oct 31 12:55:47.440367 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.440323    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d224a67-a771-46d2-ad68-c8970ca2dffa" path="/var/lib/kubelet/pods/1d224a67-a771-46d2-ad68-c8970ca2dffa/volumes"
Oct 31 12:55:47.724615 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.724590    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:55:47.735413 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.735384    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-istio-token\" (UniqueName: \"kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-istio-token\") pod \"9780b010-8c66-4490-8f1a-82296b85235e\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") "
Oct 31 12:55:47.735537 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.735518    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-data\") pod \"9780b010-8c66-4490-8f1a-82296b85235e\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") "
Oct 31 12:55:47.735582 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.735553    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-envoy\") pod \"9780b010-8c66-4490-8f1a-82296b85235e\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") "
Oct 31 12:55:47.735618 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.735584    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9780b010-8c66-4490-8f1a-82296b85235e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9780b010-8c66-4490-8f1a-82296b85235e-istiod-ca-cert\") pod \"9780b010-8c66-4490-8f1a-82296b85235e\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") "
Oct 31 12:55:47.735659 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.735643    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-socket\") pod \"9780b010-8c66-4490-8f1a-82296b85235e\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") "
Oct 31 12:55:47.735702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.735683    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-kube-api-access-8rf22\" (UniqueName: \"kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-kube-api-access-8rf22\") pod \"9780b010-8c66-4490-8f1a-82296b85235e\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") "
Oct 31 12:55:47.735777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.735756    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9780b010-8c66-4490-8f1a-82296b85235e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9780b010-8c66-4490-8f1a-82296b85235e-istio-podinfo\") pod \"9780b010-8c66-4490-8f1a-82296b85235e\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") "
Oct 31 12:55:47.735858 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.735842    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-certs\") pod \"9780b010-8c66-4490-8f1a-82296b85235e\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") "
Oct 31 12:55:47.736437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.736414    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9780b010-8c66-4490-8f1a-82296b85235e-istiod-ca-cert" pod "9780b010-8c66-4490-8f1a-82296b85235e" (UID: "9780b010-8c66-4490-8f1a-82296b85235e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:47.736749 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.736657    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9780b010-8c66-4490-8f1a-82296b85235e-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.737291 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.737254    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-socket" pod "9780b010-8c66-4490-8f1a-82296b85235e" (UID: "9780b010-8c66-4490-8f1a-82296b85235e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.737610 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.737555    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-data" pod "9780b010-8c66-4490-8f1a-82296b85235e" (UID: "9780b010-8c66-4490-8f1a-82296b85235e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.737772 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.737751    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-certs" pod "9780b010-8c66-4490-8f1a-82296b85235e" (UID: "9780b010-8c66-4490-8f1a-82296b85235e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.739192 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.739153    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-kube-api-access-8rf22" pod "9780b010-8c66-4490-8f1a-82296b85235e" (UID: "9780b010-8c66-4490-8f1a-82296b85235e"). InnerVolumeSpecName "kube-api-access-8rf22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:47.739832 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.739806    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-istio-token" pod "9780b010-8c66-4490-8f1a-82296b85235e" (UID: "9780b010-8c66-4490-8f1a-82296b85235e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:47.740026 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.740002    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-envoy" pod "9780b010-8c66-4490-8f1a-82296b85235e" (UID: "9780b010-8c66-4490-8f1a-82296b85235e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.741015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.740991    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9780b010-8c66-4490-8f1a-82296b85235e-istio-podinfo" pod "9780b010-8c66-4490-8f1a-82296b85235e" (UID: "9780b010-8c66-4490-8f1a-82296b85235e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:55:47.764586 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.764557    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:55:47.837764 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.837721    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-socket\") pod \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") "
Oct 31 12:55:47.837936 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.837780    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-token\" (UniqueName: \"kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-token\") pod \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") "
Oct 31 12:55:47.837992 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.837961    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istiod-ca-cert\") pod \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") "
Oct 31 12:55:47.838037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.837988    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-socket" pod "2054e1e9-a58c-4fe9-806c-64959e42a5d4" (UID: "2054e1e9-a58c-4fe9-806c-64959e42a5d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.838037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.837993    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-podinfo\") pod \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") "
Oct 31 12:55:47.838143 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.838042    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-credential-socket\") pod \"9780b010-8c66-4490-8f1a-82296b85235e\" (UID: \"9780b010-8c66-4490-8f1a-82296b85235e\") "
Oct 31 12:55:47.838143 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.838099    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-data\") pod \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") "
Oct 31 12:55:47.838143 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.838138    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-envoy\") pod \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") "
Oct 31 12:55:47.838307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.838170    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-kube-api-access-cbfr2\" (UniqueName: \"kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-kube-api-access-cbfr2\") pod \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") "
Oct 31 12:55:47.838307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.838231    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-certs\") pod \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") "
Oct 31 12:55:47.838307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.838267    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-credential-socket\") pod \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\" (UID: \"2054e1e9-a58c-4fe9-806c-64959e42a5d4\") "
Oct 31 12:55:47.838487 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.838346    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-data" pod "2054e1e9-a58c-4fe9-806c-64959e42a5d4" (UID: "2054e1e9-a58c-4fe9-806c-64959e42a5d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.838487 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.838472    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istiod-ca-cert" pod "2054e1e9-a58c-4fe9-806c-64959e42a5d4" (UID: "2054e1e9-a58c-4fe9-806c-64959e42a5d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:55:47.838600 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.838576    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-certs" pod "2054e1e9-a58c-4fe9-806c-64959e42a5d4" (UID: "2054e1e9-a58c-4fe9-806c-64959e42a5d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.838888 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.838864    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-credential-socket" pod "2054e1e9-a58c-4fe9-806c-64959e42a5d4" (UID: "2054e1e9-a58c-4fe9-806c-64959e42a5d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.839013 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.838862    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-credential-socket" pod "9780b010-8c66-4490-8f1a-82296b85235e" (UID: "9780b010-8c66-4490-8f1a-82296b85235e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.840254 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840213    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.840254 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840247    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.840526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840262    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.840526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840277    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.840526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840291    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.840526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840307    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.840526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840321    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.840526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840334    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.840526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840392    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.840526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840409    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rf22\" (UniqueName: \"kubernetes.io/projected/9780b010-8c66-4490-8f1a-82296b85235e-kube-api-access-8rf22\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.840526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840424    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.840526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840438    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9780b010-8c66-4490-8f1a-82296b85235e-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.840526 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.840455    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780b010-8c66-4490-8f1a-82296b85235e-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.841068 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.841042    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-envoy" pod "2054e1e9-a58c-4fe9-806c-64959e42a5d4" (UID: "2054e1e9-a58c-4fe9-806c-64959e42a5d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:55:47.841068 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.841060    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-podinfo" pod "2054e1e9-a58c-4fe9-806c-64959e42a5d4" (UID: "2054e1e9-a58c-4fe9-806c-64959e42a5d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:55:47.841192 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.841094    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-token" pod "2054e1e9-a58c-4fe9-806c-64959e42a5d4" (UID: "2054e1e9-a58c-4fe9-806c-64959e42a5d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:47.841192 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.841113    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-kube-api-access-cbfr2" pod "2054e1e9-a58c-4fe9-806c-64959e42a5d4" (UID: "2054e1e9-a58c-4fe9-806c-64959e42a5d4"). InnerVolumeSpecName "kube-api-access-cbfr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:55:47.942092 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.941999    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.942092 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.942037    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.942092 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.942054    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2054e1e9-a58c-4fe9-806c-64959e42a5d4-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:47.942092 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:47.942070    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbfr2\" (UniqueName: \"kubernetes.io/projected/2054e1e9-a58c-4fe9-806c-64959e42a5d4-kube-api-access-cbfr2\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:55:48.286949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.286868    5663 generic.go:356] "Generic (PLEG): container finished" podID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" containerID="0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394" exitCode=0
Oct 31 12:55:48.286949 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.286917    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"
Oct 31 12:55:48.287189 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.286949    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" event={"ID":"2054e1e9-a58c-4fe9-806c-64959e42a5d4","Type":"ContainerDied","Data":"0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394"}
Oct 31 12:55:48.287189 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.287006    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb" event={"ID":"2054e1e9-a58c-4fe9-806c-64959e42a5d4","Type":"ContainerDied","Data":"5cb1cdf861b0be26dd28471b635f2e106536777588598bc1c1dff477cd836c27"}
Oct 31 12:55:48.287189 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.287036    5663 scope.go:122] "RemoveContainer" containerID="0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394"
Oct 31 12:55:48.288556 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.288534    5663 generic.go:356] "Generic (PLEG): container finished" podID="9780b010-8c66-4490-8f1a-82296b85235e" containerID="0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83" exitCode=0
Oct 31 12:55:48.288670 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.288569    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" event={"ID":"9780b010-8c66-4490-8f1a-82296b85235e","Type":"ContainerDied","Data":"0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83"}
Oct 31 12:55:48.288670 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.288593    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"
Oct 31 12:55:48.288670 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.288603    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn" event={"ID":"9780b010-8c66-4490-8f1a-82296b85235e","Type":"ContainerDied","Data":"613fa47f07f6829860be78659f2ae71ca922db1acf93941f63897107fc420159"}
Oct 31 12:55:48.299129 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.299110    5663 scope.go:122] "RemoveContainer" containerID="0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394"
Oct 31 12:55:48.299721 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:55:48.299448    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394\": container with ID starting with 0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394 not found: ID does not exist" containerID="0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394"
Oct 31 12:55:48.299721 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.299482    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394"} err="failed to get container status \"0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394\": rpc error: code = NotFound desc = could not find container \"0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394\": container with ID starting with 0db138e2e407508fb90a5ca53539883445c7e4c9c8cfadd1453e415db2f03394 not found: ID does not exist"
Oct 31 12:55:48.299721 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.299505    5663 scope.go:122] "RemoveContainer" containerID="0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83"
Oct 31 12:55:48.308731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.308712    5663 scope.go:122] "RemoveContainer" containerID="0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83"
Oct 31 12:55:48.309082 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:55:48.309060    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83\": container with ID starting with 0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83 not found: ID does not exist" containerID="0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83"
Oct 31 12:55:48.309157 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.309090    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83"} err="failed to get container status \"0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83\": rpc error: code = NotFound desc = could not find container \"0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83\": container with ID starting with 0f6dd0f56c283362d2e523e206342a6364449273a76181f74958bb3c43f3ed83 not found: ID does not exist"
Oct 31 12:55:48.318061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.318038    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"]
Oct 31 12:55:48.324844 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.324824    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-rcvqb"]
Oct 31 12:55:48.357141 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.357110    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"]
Oct 31 12:55:48.361625 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:48.361600    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e6c79d-openshift-default-6859fcb4c7-lbdnn"]
Oct 31 12:55:49.033650 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:49.033566    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx" podUID="3dc43894-bca9-4020-a8a6-923731346094" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:55:49.440299 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:49.440261    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" path="/var/lib/kubelet/pods/2054e1e9-a58c-4fe9-806c-64959e42a5d4/volumes"
Oct 31 12:55:49.441006 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:49.440985    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9780b010-8c66-4490-8f1a-82296b85235e" path="/var/lib/kubelet/pods/9780b010-8c66-4490-8f1a-82296b85235e/volumes"
Oct 31 12:55:52.478125 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:55:52.478089    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:55:58.968363 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:58.968329    5663 scope.go:122] "RemoveContainer" containerID="c6fd56c85e98da1c1b89d98ffc4d26df21702bf17dd3bff3969e398a4085b0a5"
Oct 31 12:55:59.032592 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:55:59.032564    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx" podUID="3dc43894-bca9-4020-a8a6-923731346094" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:56:03.229107 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:03.229070    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"]
Oct 31 12:56:03.229596 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:03.229273    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy" containerID="cri-o://ed5f71d3b2bc6deb51b980bdf7c6454fd560f9773e5b9d1447c5ef06f6e995d0" gracePeriod=30
Oct 31 12:56:07.682501 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:07.682447    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"
Oct 31 12:56:07.792170 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:07.792126    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dc43894-bca9-4020-a8a6-923731346094-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc43894-bca9-4020-a8a6-923731346094-plugins-config-volume\") pod \"3dc43894-bca9-4020-a8a6-923731346094\" (UID: \"3dc43894-bca9-4020-a8a6-923731346094\") "
Oct 31 12:56:07.792490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:07.792461    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc43894-bca9-4020-a8a6-923731346094-kube-api-access-cqkhs\" (UniqueName: \"kubernetes.io/projected/3dc43894-bca9-4020-a8a6-923731346094-kube-api-access-cqkhs\") pod \"3dc43894-bca9-4020-a8a6-923731346094\" (UID: \"3dc43894-bca9-4020-a8a6-923731346094\") "
Oct 31 12:56:07.792672 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:07.792579    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc43894-bca9-4020-a8a6-923731346094-plugins-config-volume" pod "3dc43894-bca9-4020-a8a6-923731346094" (UID: "3dc43894-bca9-4020-a8a6-923731346094"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:56:07.793083 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:07.793060    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc43894-bca9-4020-a8a6-923731346094-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:07.795322 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:07.795291    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc43894-bca9-4020-a8a6-923731346094-kube-api-access-cqkhs" pod "3dc43894-bca9-4020-a8a6-923731346094" (UID: "3dc43894-bca9-4020-a8a6-923731346094"). InnerVolumeSpecName "kube-api-access-cqkhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:07.894190 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:07.894150    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqkhs\" (UniqueName: \"kubernetes.io/projected/3dc43894-bca9-4020-a8a6-923731346094-kube-api-access-cqkhs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.480075 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.480042    5663 generic.go:356] "Generic (PLEG): container finished" podID="22e10166-45ce-4baa-97b9-6b1544701e43" containerID="ed5f71d3b2bc6deb51b980bdf7c6454fd560f9773e5b9d1447c5ef06f6e995d0" exitCode=0
Oct 31 12:56:08.480256 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.480196    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" event={"ID":"22e10166-45ce-4baa-97b9-6b1544701e43","Type":"ContainerDied","Data":"ed5f71d3b2bc6deb51b980bdf7c6454fd560f9773e5b9d1447c5ef06f6e995d0"}
Oct 31 12:56:08.480324 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.480255    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg" event={"ID":"22e10166-45ce-4baa-97b9-6b1544701e43","Type":"ContainerDied","Data":"7d73e5d0b4e42d76f2164d9ba719c9172037824eca1837eee8f0838e952c61d8"}
Oct 31 12:56:08.480324 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.480279    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d73e5d0b4e42d76f2164d9ba719c9172037824eca1837eee8f0838e952c61d8"
Oct 31 12:56:08.480324 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.480299    5663 scope.go:122] "RemoveContainer" containerID="06624e83e2d7d3ec10dda92fc212e3731fe48d9f34fd1cb2b767bf0b148c6150"
Oct 31 12:56:08.485291 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.483084    5663 generic.go:356] "Generic (PLEG): container finished" podID="3dc43894-bca9-4020-a8a6-923731346094" containerID="fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807" exitCode=0
Oct 31 12:56:08.485291 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.483191    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx" event={"ID":"3dc43894-bca9-4020-a8a6-923731346094","Type":"ContainerDied","Data":"fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807"}
Oct 31 12:56:08.485291 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.483239    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx" event={"ID":"3dc43894-bca9-4020-a8a6-923731346094","Type":"ContainerDied","Data":"ff957a85d997ddb0609d099445a63ac75c77005a0fdedd53d009e828914317bf"}
Oct 31 12:56:08.485291 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.483341    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"
Oct 31 12:56:08.506876 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.506856    5663 scope.go:122] "RemoveContainer" containerID="fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807"
Oct 31 12:56:08.511493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.511470    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:56:08.516325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.516209    5663 scope.go:122] "RemoveContainer" containerID="fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807"
Oct 31 12:56:08.516537 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:56:08.516515    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807\": container with ID starting with fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807 not found: ID does not exist" containerID="fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807"
Oct 31 12:56:08.516611 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.516544    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807"} err="failed to get container status \"fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807\": rpc error: code = NotFound desc = could not find container \"fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807\": container with ID starting with fe838a5d5409afad07350599846268e7e19dc0b2a3c347c433f14ae09da0c807 not found: ID does not exist"
Oct 31 12:56:08.523029 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.523006    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"]
Oct 31 12:56:08.527852 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.527830    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-015701-b497b4869-jfkcx"]
Oct 31 12:56:08.605548 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.605515    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-envoy\") pod \"22e10166-45ce-4baa-97b9-6b1544701e43\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") "
Oct 31 12:56:08.605723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.605600    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-data\") pod \"22e10166-45ce-4baa-97b9-6b1544701e43\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") "
Oct 31 12:56:08.605723 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.605659    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-certs\") pod \"22e10166-45ce-4baa-97b9-6b1544701e43\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") "
Oct 31 12:56:08.605864 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.605730    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e10166-45ce-4baa-97b9-6b1544701e43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e10166-45ce-4baa-97b9-6b1544701e43-istiod-ca-cert\") pod \"22e10166-45ce-4baa-97b9-6b1544701e43\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") "
Oct 31 12:56:08.605864 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.605799    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-credential-socket\") pod \"22e10166-45ce-4baa-97b9-6b1544701e43\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") "
Oct 31 12:56:08.605864 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.605825    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22e10166-45ce-4baa-97b9-6b1544701e43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e10166-45ce-4baa-97b9-6b1544701e43-istio-podinfo\") pod \"22e10166-45ce-4baa-97b9-6b1544701e43\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") "
Oct 31 12:56:08.605864 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.605849    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-istio-token\" (UniqueName: \"kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-istio-token\") pod \"22e10166-45ce-4baa-97b9-6b1544701e43\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") "
Oct 31 12:56:08.606091 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.605886    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-kube-api-access-6dzjt\" (UniqueName: \"kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-kube-api-access-6dzjt\") pod \"22e10166-45ce-4baa-97b9-6b1544701e43\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") "
Oct 31 12:56:08.606091 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.605906    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-data" pod "22e10166-45ce-4baa-97b9-6b1544701e43" (UID: "22e10166-45ce-4baa-97b9-6b1544701e43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:08.606091 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.605940    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-socket\") pod \"22e10166-45ce-4baa-97b9-6b1544701e43\" (UID: \"22e10166-45ce-4baa-97b9-6b1544701e43\") "
Oct 31 12:56:08.606091 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.605977    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-certs" pod "22e10166-45ce-4baa-97b9-6b1544701e43" (UID: "22e10166-45ce-4baa-97b9-6b1544701e43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:08.606091 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.606080    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-credential-socket" pod "22e10166-45ce-4baa-97b9-6b1544701e43" (UID: "22e10166-45ce-4baa-97b9-6b1544701e43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:08.606349 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.606259    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-socket" pod "22e10166-45ce-4baa-97b9-6b1544701e43" (UID: "22e10166-45ce-4baa-97b9-6b1544701e43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:08.606349 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.606290    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e10166-45ce-4baa-97b9-6b1544701e43-istiod-ca-cert" pod "22e10166-45ce-4baa-97b9-6b1544701e43" (UID: "22e10166-45ce-4baa-97b9-6b1544701e43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:56:08.606767 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.606739    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.606892 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.606769    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.606892 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.606784    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.606892 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.606799    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e10166-45ce-4baa-97b9-6b1544701e43-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.606892 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.606813    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.607910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.607876    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-envoy" pod "22e10166-45ce-4baa-97b9-6b1544701e43" (UID: "22e10166-45ce-4baa-97b9-6b1544701e43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:08.608165 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.608142    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-kube-api-access-6dzjt" pod "22e10166-45ce-4baa-97b9-6b1544701e43" (UID: "22e10166-45ce-4baa-97b9-6b1544701e43"). InnerVolumeSpecName "kube-api-access-6dzjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:08.608266 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.608173    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-istio-token" pod "22e10166-45ce-4baa-97b9-6b1544701e43" (UID: "22e10166-45ce-4baa-97b9-6b1544701e43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:08.608266 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.608235    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22e10166-45ce-4baa-97b9-6b1544701e43-istio-podinfo" pod "22e10166-45ce-4baa-97b9-6b1544701e43" (UID: "22e10166-45ce-4baa-97b9-6b1544701e43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:56:08.708628 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.708592    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e10166-45ce-4baa-97b9-6b1544701e43-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.708628 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.708621    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.708628 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.708632    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dzjt\" (UniqueName: \"kubernetes.io/projected/22e10166-45ce-4baa-97b9-6b1544701e43-kube-api-access-6dzjt\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:08.709072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:08.708642    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e10166-45ce-4baa-97b9-6b1544701e43-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:09.440896 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:09.440860    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc43894-bca9-4020-a8a6-923731346094" path="/var/lib/kubelet/pods/3dc43894-bca9-4020-a8a6-923731346094/volumes"
Oct 31 12:56:09.492165 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:09.492139    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"
Oct 31 12:56:09.511223 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:09.511184    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"]
Oct 31 12:56:09.515548 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:09.515521    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-015701-openshift-default-86f6dfd4b7-qxbzg"]
Oct 31 12:56:11.440025 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:11.439990    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" path="/var/lib/kubelet/pods/22e10166-45ce-4baa-97b9-6b1544701e43/volumes"
Oct 31 12:56:22.314650 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.314613    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"]
Oct 31 12:56:22.316177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316146    5663 cpu_manager.go:414] "containerMap: removing container" podUID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" containerName="istio-proxy"
Oct 31 12:56:22.316177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316174    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" containerName="istio-proxy"
Oct 31 12:56:22.316472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316209    5663 cpu_manager.go:414] "containerMap: removing container" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy"
Oct 31 12:56:22.316472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316217    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy"
Oct 31 12:56:22.316472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316237    5663 cpu_manager.go:414] "containerMap: removing container" podUID="9780b010-8c66-4490-8f1a-82296b85235e" containerName="istio-proxy"
Oct 31 12:56:22.316472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316245    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9780b010-8c66-4490-8f1a-82296b85235e" containerName="istio-proxy"
Oct 31 12:56:22.316472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316259    5663 cpu_manager.go:414] "containerMap: removing container" podUID="3dc43894-bca9-4020-a8a6-923731346094" containerName="payload-processing"
Oct 31 12:56:22.316472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316266    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc43894-bca9-4020-a8a6-923731346094" containerName="payload-processing"
Oct 31 12:56:22.316472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316300    5663 cpu_manager.go:414] "containerMap: removing container" podUID="1d224a67-a771-46d2-ad68-c8970ca2dffa" containerName="main"
Oct 31 12:56:22.316472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316309    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d224a67-a771-46d2-ad68-c8970ca2dffa" containerName="main"
Oct 31 12:56:22.316968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316584    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy"
Oct 31 12:56:22.316968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316603    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="9780b010-8c66-4490-8f1a-82296b85235e" containerName="istio-proxy"
Oct 31 12:56:22.316968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316611    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc43894-bca9-4020-a8a6-923731346094" containerName="payload-processing"
Oct 31 12:56:22.316968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316623    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d224a67-a771-46d2-ad68-c8970ca2dffa" containerName="main"
Oct 31 12:56:22.316968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316631    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy"
Oct 31 12:56:22.316968 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.316637    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="2054e1e9-a58c-4fe9-806c-64959e42a5d4" containerName="istio-proxy"
Oct 31 12:56:22.321183 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.321158    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.331646 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.331602    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"]
Oct 31 12:56:22.334924 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.334885    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.335073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.334932    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph26k\" (UniqueName: \"kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-kube-api-access-ph26k\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.335073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.334990    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.335073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.335013    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16395ca3-67a4-48e6-8859-a941d5bec3f3-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.335073 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.335039    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.335310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.335103    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.335310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.335241    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.335310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.335278    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.335497 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.335344    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.436530 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.436494    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.436722 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.436547    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.436722 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.436604    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.436722 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.436714    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.436914 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.436741    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph26k\" (UniqueName: \"kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-kube-api-access-ph26k\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.436914 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.436798    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.436914 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.436841    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16395ca3-67a4-48e6-8859-a941d5bec3f3-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.436914 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.436870    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.437123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.436937    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.437123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.437002    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-certs\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.437123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.437049    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-credential-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.437123 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.437112    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-data\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.437463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.437438    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-socket\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.437691 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.437667    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16395ca3-67a4-48e6-8859-a941d5bec3f3-istiod-ca-cert\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.439260 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.439232    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-envoy\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.439390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.439373    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-podinfo\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.449146 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.449121    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-token\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.451567 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.451539    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph26k\" (UniqueName: \"kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-kube-api-access-ph26k\") pod \"e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") " pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.477413 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:56:22.477335    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:56:22.647461 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.647321    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:22.950529 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.950495    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"]
Oct 31 12:56:22.952553 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:56:22.952522    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16395ca3_67a4_48e6_8859_a941d5bec3f3.slice/crio-ffce4ac63845bc72480ec9cac505625c902b4b333fb64830dfe842f9af11a0ba WatchSource:0}: Error finding container ffce4ac63845bc72480ec9cac505625c902b4b333fb64830dfe842f9af11a0ba: Status 404 returned error can't find the container with id ffce4ac63845bc72480ec9cac505625c902b4b333fb64830dfe842f9af11a0ba
Oct 31 12:56:22.955587 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.955538    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:56:22.955695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.955624    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:56:22.955695 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:22.955659    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 12:56:23.596808 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:23.596768    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" event={"ID":"16395ca3-67a4-48e6-8859-a941d5bec3f3","Type":"ContainerStarted","Data":"4b9fa27c813345e7214e9b95bd02d97de6bbc1108ff8e8ffdb95ef6415fd3859"}
Oct 31 12:56:23.597427 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:23.596840    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" event={"ID":"16395ca3-67a4-48e6-8859-a941d5bec3f3","Type":"ContainerStarted","Data":"ffce4ac63845bc72480ec9cac505625c902b4b333fb64830dfe842f9af11a0ba"}
Oct 31 12:56:23.630048 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:23.630000    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" podStartSLOduration=1.629985328 podStartE2EDuration="1.629985328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:23.627290757 +0000 UTC m=+1708.367716972" watchObservedRunningTime="2026-10-31 12:56:23.629985328 +0000 UTC m=+1708.370411616"
Oct 31 12:56:23.648151 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:23.648124    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:24.596992 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:24.594533    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"]
Oct 31 12:56:24.596992 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:24.594803    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf" podUID="330b17f2-d442-4686-aec9-81654f1d01b2" containerName="maas-api" containerID="cri-o://b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3" gracePeriod=30
Oct 31 12:56:24.671665 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:24.671631    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:56:24.671957 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:24.671925    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" podUID="16395ca3-67a4-48e6-8859-a941d5bec3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 12:56:25.280377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.274708    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:56:25.416521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.416492    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-metrics-certs\") pod \"330b17f2-d442-4686-aec9-81654f1d01b2\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") "
Oct 31 12:56:25.416774 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.416749    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330b17f2-d442-4686-aec9-81654f1d01b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/330b17f2-d442-4686-aec9-81654f1d01b2-openshift-service-ca\") pod \"330b17f2-d442-4686-aec9-81654f1d01b2\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") "
Oct 31 12:56:25.417066 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.417043    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330b17f2-d442-4686-aec9-81654f1d01b2-kube-api-access-r988s\" (UniqueName: \"kubernetes.io/projected/330b17f2-d442-4686-aec9-81654f1d01b2-kube-api-access-r988s\") pod \"330b17f2-d442-4686-aec9-81654f1d01b2\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") "
Oct 31 12:56:25.417162 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.417086    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls\") pod \"330b17f2-d442-4686-aec9-81654f1d01b2\" (UID: \"330b17f2-d442-4686-aec9-81654f1d01b2\") "
Oct 31 12:56:25.417162 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.417086    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330b17f2-d442-4686-aec9-81654f1d01b2-openshift-service-ca" pod "330b17f2-d442-4686-aec9-81654f1d01b2" (UID: "330b17f2-d442-4686-aec9-81654f1d01b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:56:25.417700 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.417663    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/330b17f2-d442-4686-aec9-81654f1d01b2-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:25.419389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.419336    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-metrics-certs" pod "330b17f2-d442-4686-aec9-81654f1d01b2" (UID: "330b17f2-d442-4686-aec9-81654f1d01b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:56:25.425738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.422836    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls" pod "330b17f2-d442-4686-aec9-81654f1d01b2" (UID: "330b17f2-d442-4686-aec9-81654f1d01b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:56:25.425883 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.422980    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330b17f2-d442-4686-aec9-81654f1d01b2-kube-api-access-r988s" pod "330b17f2-d442-4686-aec9-81654f1d01b2" (UID: "330b17f2-d442-4686-aec9-81654f1d01b2"). InnerVolumeSpecName "kube-api-access-r988s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:25.518915 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.518831    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r988s\" (UniqueName: \"kubernetes.io/projected/330b17f2-d442-4686-aec9-81654f1d01b2-kube-api-access-r988s\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:25.518915 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.518861    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:25.518915 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.518873    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/330b17f2-d442-4686-aec9-81654f1d01b2-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:25.629771 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.629735    5663 generic.go:356] "Generic (PLEG): container finished" podID="330b17f2-d442-4686-aec9-81654f1d01b2" containerID="b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3" exitCode=0
Oct 31 12:56:25.630247 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.629779    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"
Oct 31 12:56:25.630247 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.629814    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf" event={"ID":"330b17f2-d442-4686-aec9-81654f1d01b2","Type":"ContainerDied","Data":"b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3"}
Oct 31 12:56:25.630247 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.629867    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf" event={"ID":"330b17f2-d442-4686-aec9-81654f1d01b2","Type":"ContainerDied","Data":"bc338bf6ae2d951594172c37d6322ad152fdb58f902cddc755495cdd7047752e"}
Oct 31 12:56:25.630247 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.629893    5663 scope.go:122] "RemoveContainer" containerID="b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3"
Oct 31 12:56:25.639811 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.639794    5663 scope.go:122] "RemoveContainer" containerID="b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3"
Oct 31 12:56:25.640065 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:56:25.640046    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3\": container with ID starting with b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3 not found: ID does not exist" containerID="b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3"
Oct 31 12:56:25.640127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.640073    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3"} err="failed to get container status \"b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3\": rpc error: code = NotFound desc = could not find container \"b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3\": container with ID starting with b387356c4b348b464642ac4b3a73aa3ba6454c8225bfbf5c43d9c5e14c542ad3 not found: ID does not exist"
Oct 31 12:56:25.648901 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.648874    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:56:25.649018 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.648921    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" podUID="16395ca3-67a4-48e6-8859-a941d5bec3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 12:56:25.654766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.650383    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"]
Oct 31 12:56:25.672376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:25.656959    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ad1af7-875478959-h5kqf"]
Oct 31 12:56:26.647956 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:26.647927    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:56:26.648502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:26.647982    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" podUID="16395ca3-67a4-48e6-8859-a941d5bec3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 12:56:27.439532 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:27.439499    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330b17f2-d442-4686-aec9-81654f1d01b2" path="/var/lib/kubelet/pods/330b17f2-d442-4686-aec9-81654f1d01b2/volumes"
Oct 31 12:56:27.647704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:27.647668    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:56:27.647909 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:27.647740    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" podUID="16395ca3-67a4-48e6-8859-a941d5bec3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 12:56:28.647776 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:28.647746    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:56:28.648201 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:28.647795    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" podUID="16395ca3-67a4-48e6-8859-a941d5bec3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 12:56:29.647622 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:29.647583    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 12:56:29.647804 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:29.647642    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" podUID="16395ca3-67a4-48e6-8859-a941d5bec3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 12:56:30.668730 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:30.668688    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:30.678872 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:30.678842    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:30.684738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:30.683630    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:56:33.231905 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:33.231807    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"]
Oct 31 12:56:33.232415 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:33.232097    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy" containerID="cri-o://50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250" gracePeriod=30
Oct 31 12:56:33.951825 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:33.951793    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"]
Oct 31 12:56:33.952409 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:33.952394    5663 cpu_manager.go:414] "containerMap: removing container" podUID="330b17f2-d442-4686-aec9-81654f1d01b2" containerName="maas-api"
Oct 31 12:56:33.952479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:33.952409    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330b17f2-d442-4686-aec9-81654f1d01b2" containerName="maas-api"
Oct 31 12:56:33.952479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:33.952429    5663 cpu_manager.go:414] "containerMap: removing container" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy"
Oct 31 12:56:33.952479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:33.952434    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e10166-45ce-4baa-97b9-6b1544701e43" containerName="istio-proxy"
Oct 31 12:56:33.952580 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:33.952542    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="330b17f2-d442-4686-aec9-81654f1d01b2" containerName="maas-api"
Oct 31 12:56:33.957182 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:33.957165    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:33.965372 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:33.965162    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:56:33.965372 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:33.965326    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:56:33.968871 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:33.968847    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"]
Oct 31 12:56:34.093770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:34.093735    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgmwd\" (UniqueName: \"kubernetes.io/projected/180ad758-df17-403d-9112-aeb3063f288a-kube-api-access-jgmwd\") pod \"maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx\" (UID: \"180ad758-df17-403d-9112-aeb3063f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:34.093941 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:34.093804    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180ad758-df17-403d-9112-aeb3063f288a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx\" (UID: \"180ad758-df17-403d-9112-aeb3063f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:34.094015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:34.093979    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180ad758-df17-403d-9112-aeb3063f288a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx\" (UID: \"180ad758-df17-403d-9112-aeb3063f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:34.195462 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:34.195419    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180ad758-df17-403d-9112-aeb3063f288a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx\" (UID: \"180ad758-df17-403d-9112-aeb3063f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:34.195623 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:34.195598    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180ad758-df17-403d-9112-aeb3063f288a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx\" (UID: \"180ad758-df17-403d-9112-aeb3063f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:34.195689 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:34.195673    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgmwd\" (UniqueName: \"kubernetes.io/projected/180ad758-df17-403d-9112-aeb3063f288a-kube-api-access-jgmwd\") pod \"maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx\" (UID: \"180ad758-df17-403d-9112-aeb3063f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:34.196072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:34.196049    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180ad758-df17-403d-9112-aeb3063f288a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx\" (UID: \"180ad758-df17-403d-9112-aeb3063f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:34.198071 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:34.198037    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180ad758-df17-403d-9112-aeb3063f288a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx\" (UID: \"180ad758-df17-403d-9112-aeb3063f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:34.208330 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:34.208269    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgmwd\" (UniqueName: \"kubernetes.io/projected/180ad758-df17-403d-9112-aeb3063f288a-kube-api-access-jgmwd\") pod \"maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx\" (UID: \"180ad758-df17-403d-9112-aeb3063f288a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:34.277846 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:34.277815    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:34.562583 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:34.562546    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"]
Oct 31 12:56:34.563598 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:56:34.563571    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180ad758_df17_403d_9112_aeb3063f288a.slice/crio-71f661cec941204ab0fdf47cb166c9774c21f01fd3115f912e308b78873a5fb7 WatchSource:0}: Error finding container 71f661cec941204ab0fdf47cb166c9774c21f01fd3115f912e308b78873a5fb7: Status 404 returned error can't find the container with id 71f661cec941204ab0fdf47cb166c9774c21f01fd3115f912e308b78873a5fb7
Oct 31 12:56:34.746554 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:34.746517    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx" event={"ID":"180ad758-df17-403d-9112-aeb3063f288a","Type":"ContainerStarted","Data":"71f661cec941204ab0fdf47cb166c9774c21f01fd3115f912e308b78873a5fb7"}
Oct 31 12:56:35.754922 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:35.754883    5663 generic.go:356] "Generic (PLEG): container finished" podID="180ad758-df17-403d-9112-aeb3063f288a" containerID="b965834faeee4d9f24f1f99a5f504b4330318a5a5629e60fd97ae7b67ef5d7e0" exitCode=0
Oct 31 12:56:35.755298 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:35.754954    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx" event={"ID":"180ad758-df17-403d-9112-aeb3063f288a","Type":"ContainerDied","Data":"b965834faeee4d9f24f1f99a5f504b4330318a5a5629e60fd97ae7b67ef5d7e0"}
Oct 31 12:56:36.916675 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:36.916648    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:36.935610 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:36.935569    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/180ad758-df17-403d-9112-aeb3063f288a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180ad758-df17-403d-9112-aeb3063f288a-maas-api-openshift-ca\") pod \"180ad758-df17-403d-9112-aeb3063f288a\" (UID: \"180ad758-df17-403d-9112-aeb3063f288a\") "
Oct 31 12:56:36.935765 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:36.935630    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180ad758-df17-403d-9112-aeb3063f288a-kube-api-access-jgmwd\" (UniqueName: \"kubernetes.io/projected/180ad758-df17-403d-9112-aeb3063f288a-kube-api-access-jgmwd\") pod \"180ad758-df17-403d-9112-aeb3063f288a\" (UID: \"180ad758-df17-403d-9112-aeb3063f288a\") "
Oct 31 12:56:36.935765 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:36.935728    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/180ad758-df17-403d-9112-aeb3063f288a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180ad758-df17-403d-9112-aeb3063f288a-maas-api-xks-ca\") pod \"180ad758-df17-403d-9112-aeb3063f288a\" (UID: \"180ad758-df17-403d-9112-aeb3063f288a\") "
Oct 31 12:56:36.936612 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:36.936583    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180ad758-df17-403d-9112-aeb3063f288a-maas-api-openshift-ca" pod "180ad758-df17-403d-9112-aeb3063f288a" (UID: "180ad758-df17-403d-9112-aeb3063f288a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:56:36.938740 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:36.938712    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180ad758-df17-403d-9112-aeb3063f288a-maas-api-xks-ca" pod "180ad758-df17-403d-9112-aeb3063f288a" (UID: "180ad758-df17-403d-9112-aeb3063f288a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:56:36.940255 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:36.940226    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180ad758-df17-403d-9112-aeb3063f288a-kube-api-access-jgmwd" pod "180ad758-df17-403d-9112-aeb3063f288a" (UID: "180ad758-df17-403d-9112-aeb3063f288a"). InnerVolumeSpecName "kube-api-access-jgmwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:37.038231 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:37.038145    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180ad758-df17-403d-9112-aeb3063f288a-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:37.038231 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:37.038178    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgmwd\" (UniqueName: \"kubernetes.io/projected/180ad758-df17-403d-9112-aeb3063f288a-kube-api-access-jgmwd\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:37.038231 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:37.038189    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180ad758-df17-403d-9112-aeb3063f288a-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:37.769986 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:37.769908    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"
Oct 31 12:56:37.769986 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:37.769937    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx" event={"ID":"180ad758-df17-403d-9112-aeb3063f288a","Type":"ContainerDied","Data":"71f661cec941204ab0fdf47cb166c9774c21f01fd3115f912e308b78873a5fb7"}
Oct 31 12:56:37.770194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:37.769995    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f661cec941204ab0fdf47cb166c9774c21f01fd3115f912e308b78873a5fb7"
Oct 31 12:56:38.530462 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.530400    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:56:38.559973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.558438    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-socket\") pod \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") "
Oct 31 12:56:38.559973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.558498    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-credential-socket\") pod \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") "
Oct 31 12:56:38.559973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.558573    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-envoy\") pod \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") "
Oct 31 12:56:38.559973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.558628    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-kube-api-access-jvkms\" (UniqueName: \"kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-kube-api-access-jvkms\") pod \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") "
Oct 31 12:56:38.559973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.558690    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-data\") pod \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") "
Oct 31 12:56:38.559973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.558781    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-podinfo\") pod \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") "
Oct 31 12:56:38.559973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.558863    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-certs\") pod \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") "
Oct 31 12:56:38.559973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.558997    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istiod-ca-cert\") pod \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") "
Oct 31 12:56:38.559973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.559044    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-token\" (UniqueName: \"kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-token\") pod \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\" (UID: \"9a7ea3d2-2730-4332-8b30-dd2400e2239d\") "
Oct 31 12:56:38.561731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.560441    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-credential-socket" pod "9a7ea3d2-2730-4332-8b30-dd2400e2239d" (UID: "9a7ea3d2-2730-4332-8b30-dd2400e2239d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:38.561731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.561168    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-socket" pod "9a7ea3d2-2730-4332-8b30-dd2400e2239d" (UID: "9a7ea3d2-2730-4332-8b30-dd2400e2239d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:38.561731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.561276    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-data" pod "9a7ea3d2-2730-4332-8b30-dd2400e2239d" (UID: "9a7ea3d2-2730-4332-8b30-dd2400e2239d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:38.561731 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.561480    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-kube-api-access-jvkms" pod "9a7ea3d2-2730-4332-8b30-dd2400e2239d" (UID: "9a7ea3d2-2730-4332-8b30-dd2400e2239d"). InnerVolumeSpecName "kube-api-access-jvkms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:38.561942 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.561851    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istiod-ca-cert" pod "9a7ea3d2-2730-4332-8b30-dd2400e2239d" (UID: "9a7ea3d2-2730-4332-8b30-dd2400e2239d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:56:38.563398 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.562045    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-certs" pod "9a7ea3d2-2730-4332-8b30-dd2400e2239d" (UID: "9a7ea3d2-2730-4332-8b30-dd2400e2239d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:38.563509 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.563454    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-podinfo" pod "9a7ea3d2-2730-4332-8b30-dd2400e2239d" (UID: "9a7ea3d2-2730-4332-8b30-dd2400e2239d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:56:38.564543 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.564515    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-envoy" pod "9a7ea3d2-2730-4332-8b30-dd2400e2239d" (UID: "9a7ea3d2-2730-4332-8b30-dd2400e2239d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:38.565770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.565743    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-token" pod "9a7ea3d2-2730-4332-8b30-dd2400e2239d" (UID: "9a7ea3d2-2730-4332-8b30-dd2400e2239d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:38.661316 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.661271    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.661316 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.661304    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.661316 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.661313    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.661316 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.661322    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.661316 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.661330    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.661652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.661405    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.661652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.661420    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.661652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.661435    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvkms\" (UniqueName: \"kubernetes.io/projected/9a7ea3d2-2730-4332-8b30-dd2400e2239d-kube-api-access-jvkms\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.661652 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.661446    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7ea3d2-2730-4332-8b30-dd2400e2239d-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:38.698201 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.698161    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"]
Oct 31 12:56:38.716000 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.715968    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"]
Oct 31 12:56:38.778591 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.778556    5663 generic.go:356] "Generic (PLEG): container finished" podID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerID="50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250" exitCode=0
Oct 31 12:56:38.778802 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.778628    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"
Oct 31 12:56:38.778802 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.778637    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" event={"ID":"9a7ea3d2-2730-4332-8b30-dd2400e2239d","Type":"ContainerDied","Data":"50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250"}
Oct 31 12:56:38.778802 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.778746    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv" event={"ID":"9a7ea3d2-2730-4332-8b30-dd2400e2239d","Type":"ContainerDied","Data":"2b11fff71ed79abbdc50927191e0b88117895c4b133fb05f48688a2b179531f9"}
Oct 31 12:56:38.778802 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.778779    5663 scope.go:122] "RemoveContainer" containerID="50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250"
Oct 31 12:56:38.791642 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.791622    5663 scope.go:122] "RemoveContainer" containerID="0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58"
Oct 31 12:56:38.802891 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.802860    5663 scope.go:122] "RemoveContainer" containerID="50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250"
Oct 31 12:56:38.803270 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:56:38.803070    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250\": container with ID starting with 50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250 not found: ID does not exist" containerID="50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250"
Oct 31 12:56:38.803270 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.803104    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250"} err="failed to get container status \"50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250\": rpc error: code = NotFound desc = could not find container \"50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250\": container with ID starting with 50a37af29deecb799ab36a835f46e1f227a2185c202fd5b364bb395afbb72250 not found: ID does not exist"
Oct 31 12:56:38.803270 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.803125    5663 scope.go:122] "RemoveContainer" containerID="0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58"
Oct 31 12:56:38.803493 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:56:38.803361    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58\": container with ID starting with 0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58 not found: ID does not exist" containerID="0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58"
Oct 31 12:56:38.803493 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.803391    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58"} err="failed to get container status \"0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58\": rpc error: code = NotFound desc = could not find container \"0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58\": container with ID starting with 0ea25689a139953b92ac852c888ce7f24a02d4b56da01c819cf39df31312ba58 not found: ID does not exist"
Oct 31 12:56:38.812423 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.812396    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"]
Oct 31 12:56:38.822998 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:38.822973    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ad1af7-openshift-default-5bbb65bdfc-zrbkv"]
Oct 31 12:56:39.439657 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:39.439628    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" path="/var/lib/kubelet/pods/9a7ea3d2-2730-4332-8b30-dd2400e2239d/volumes"
Oct 31 12:56:50.879615 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.879582    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"]
Oct 31 12:56:50.880164 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.880150    5663 cpu_manager.go:414] "containerMap: removing container" podUID="180ad758-df17-403d-9112-aeb3063f288a" containerName="revoke-keys"
Oct 31 12:56:50.880219 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.880165    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180ad758-df17-403d-9112-aeb3063f288a" containerName="revoke-keys"
Oct 31 12:56:50.880219 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.880192    5663 cpu_manager.go:414] "containerMap: removing container" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy"
Oct 31 12:56:50.880219 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.880198    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy"
Oct 31 12:56:50.880219 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.880214    5663 cpu_manager.go:414] "containerMap: removing container" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy"
Oct 31 12:56:50.880219 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.880219    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy"
Oct 31 12:56:50.880419 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.880296    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy"
Oct 31 12:56:50.880419 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.880309    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7ea3d2-2730-4332-8b30-dd2400e2239d" containerName="istio-proxy"
Oct 31 12:56:50.880419 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.880319    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="180ad758-df17-403d-9112-aeb3063f288a" containerName="revoke-keys"
Oct 31 12:56:50.883964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.883944    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:50.900076 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.896688    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:56:50.900076 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.896699    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:56:50.900076 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.899012    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"]
Oct 31 12:56:50.903015 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.902987    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vzc\" (UniqueName: \"kubernetes.io/projected/57d987d0-d800-436e-b59e-779b730a976e-kube-api-access-z6vzc\") pod \"maas-api-revoke-sub-d9badbf0-p6ljb\" (UID: \"57d987d0-d800-436e-b59e-779b730a976e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:50.903163 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.903140    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d987d0-d800-436e-b59e-779b730a976e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9badbf0-p6ljb\" (UID: \"57d987d0-d800-436e-b59e-779b730a976e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:50.903395 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:50.903371    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d987d0-d800-436e-b59e-779b730a976e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9badbf0-p6ljb\" (UID: \"57d987d0-d800-436e-b59e-779b730a976e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:51.004856 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:51.004821    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d987d0-d800-436e-b59e-779b730a976e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9badbf0-p6ljb\" (UID: \"57d987d0-d800-436e-b59e-779b730a976e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:51.005035 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:51.004874    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vzc\" (UniqueName: \"kubernetes.io/projected/57d987d0-d800-436e-b59e-779b730a976e-kube-api-access-z6vzc\") pod \"maas-api-revoke-sub-d9badbf0-p6ljb\" (UID: \"57d987d0-d800-436e-b59e-779b730a976e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:51.005035 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:51.004980    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d987d0-d800-436e-b59e-779b730a976e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9badbf0-p6ljb\" (UID: \"57d987d0-d800-436e-b59e-779b730a976e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:51.005710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:51.005690    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d987d0-d800-436e-b59e-779b730a976e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9badbf0-p6ljb\" (UID: \"57d987d0-d800-436e-b59e-779b730a976e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:51.007480 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:51.007453    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d987d0-d800-436e-b59e-779b730a976e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9badbf0-p6ljb\" (UID: \"57d987d0-d800-436e-b59e-779b730a976e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:51.018266 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:51.018236    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vzc\" (UniqueName: \"kubernetes.io/projected/57d987d0-d800-436e-b59e-779b730a976e-kube-api-access-z6vzc\") pod \"maas-api-revoke-sub-d9badbf0-p6ljb\" (UID: \"57d987d0-d800-436e-b59e-779b730a976e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:51.200312 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:51.200231    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:51.532366 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:51.532313    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"]
Oct 31 12:56:51.535330 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:56:51.535293    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d987d0_d800_436e_b59e_779b730a976e.slice/crio-63638a891d9ff543191ffcb9d5e37ace8e297c84dcd15709417988b3fc6d5017 WatchSource:0}: Error finding container 63638a891d9ff543191ffcb9d5e37ace8e297c84dcd15709417988b3fc6d5017: Status 404 returned error can't find the container with id 63638a891d9ff543191ffcb9d5e37ace8e297c84dcd15709417988b3fc6d5017
Oct 31 12:56:51.907170 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:51.907132    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb" event={"ID":"57d987d0-d800-436e-b59e-779b730a976e","Type":"ContainerStarted","Data":"f3f567526c477d0d9aef807b3fa20c5c2345389a57602dc97f5da663e708c6b9"}
Oct 31 12:56:51.907636 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:51.907186    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb" event={"ID":"57d987d0-d800-436e-b59e-779b730a976e","Type":"ContainerStarted","Data":"63638a891d9ff543191ffcb9d5e37ace8e297c84dcd15709417988b3fc6d5017"}
Oct 31 12:56:51.927197 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:51.927149    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb" podStartSLOduration=1.926697407 podStartE2EDuration="1.926697407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:56:51.925644924 +0000 UTC m=+1736.666071140" watchObservedRunningTime="2026-10-31 12:56:51.926697407 +0000 UTC m=+1736.667123628"
Oct 31 12:56:52.483433 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:56:52.483394    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:56:52.915374 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:52.915334    5663 generic.go:356] "Generic (PLEG): container finished" podID="57d987d0-d800-436e-b59e-779b730a976e" containerID="f3f567526c477d0d9aef807b3fa20c5c2345389a57602dc97f5da663e708c6b9" exitCode=0
Oct 31 12:56:52.915767 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:52.915402    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb" event={"ID":"57d987d0-d800-436e-b59e-779b730a976e","Type":"ContainerDied","Data":"f3f567526c477d0d9aef807b3fa20c5c2345389a57602dc97f5da663e708c6b9"}
Oct 31 12:56:53.721592 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:53.721553    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main" containerID="cri-o://79e58cb93e2407e0ac9f2c853560649c88d9c89f96f9303951dace223049a109" gracePeriod=45
Oct 31 12:56:53.739654 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:53.739616    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" containerID="cri-o://95d49f43d2b23fd6143ff1577ebb53575486efb1961e8ebe999bef55e6e5055d" gracePeriod=45
Oct 31 12:56:53.876077 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:53.875992    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused"
Oct 31 12:56:53.924933 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:53.924900    5663 generic.go:356] "Generic (PLEG): container finished" podID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerID="95d49f43d2b23fd6143ff1577ebb53575486efb1961e8ebe999bef55e6e5055d" exitCode=0
Oct 31 12:56:53.925441 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:53.924976    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" event={"ID":"e93d0d4e-ec94-4653-9bfb-7dd27581502a","Type":"ContainerDied","Data":"95d49f43d2b23fd6143ff1577ebb53575486efb1961e8ebe999bef55e6e5055d"}
Oct 31 12:56:53.927300 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:53.927276    5663 generic.go:356] "Generic (PLEG): container finished" podID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerID="79e58cb93e2407e0ac9f2c853560649c88d9c89f96f9303951dace223049a109" exitCode=0
Oct 31 12:56:53.927552 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:53.927521    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" event={"ID":"4eddddf5-9ee1-4d43-a506-ac57824bc0c4","Type":"ContainerDied","Data":"79e58cb93e2407e0ac9f2c853560649c88d9c89f96f9303951dace223049a109"}
Oct 31 12:56:54.085276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.085223    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:56:54.113430 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.106215    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:56:54.129252 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.129223    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:54.173232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.172712    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-home\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-home\") pod \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") "
Oct 31 12:56:54.173232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.172791    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93d0d4e-ec94-4653-9bfb-7dd27581502a-kube-api-access-z7nd6\" (UniqueName: \"kubernetes.io/projected/e93d0d4e-ec94-4653-9bfb-7dd27581502a-kube-api-access-z7nd6\") pod \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") "
Oct 31 12:56:54.173232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.172838    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tls-certs\") pod \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") "
Oct 31 12:56:54.173232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.172877    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-home\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-home\") pod \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") "
Oct 31 12:56:54.173232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.172946    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-model-cache\") pod \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") "
Oct 31 12:56:54.173232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.172993    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tmp-dir\") pod \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") "
Oct 31 12:56:54.173232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.173087    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-dshm\") pod \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") "
Oct 31 12:56:54.173232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.173179    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-dshm\") pod \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\" (UID: \"e93d0d4e-ec94-4653-9bfb-7dd27581502a\") "
Oct 31 12:56:54.174223 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.173809    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-kube-api-access-47x8n\" (UniqueName: \"kubernetes.io/projected/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-kube-api-access-47x8n\") pod \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") "
Oct 31 12:56:54.174390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.174331    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-model-cache\") pod \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") "
Oct 31 12:56:54.174508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.174493    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tmp-dir\") pod \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") "
Oct 31 12:56:54.174581 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.174535    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tls-certs\") pod \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\" (UID: \"4eddddf5-9ee1-4d43-a506-ac57824bc0c4\") "
Oct 31 12:56:54.177900 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.177584    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-home" pod "e93d0d4e-ec94-4653-9bfb-7dd27581502a" (UID: "e93d0d4e-ec94-4653-9bfb-7dd27581502a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:54.178152 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.178130    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-model-cache" pod "4eddddf5-9ee1-4d43-a506-ac57824bc0c4" (UID: "4eddddf5-9ee1-4d43-a506-ac57824bc0c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:54.178559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.178229    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-home" pod "4eddddf5-9ee1-4d43-a506-ac57824bc0c4" (UID: "4eddddf5-9ee1-4d43-a506-ac57824bc0c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:54.178559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.178310    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-dshm" pod "4eddddf5-9ee1-4d43-a506-ac57824bc0c4" (UID: "4eddddf5-9ee1-4d43-a506-ac57824bc0c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:54.178559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.178341    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tmp-dir" pod "4eddddf5-9ee1-4d43-a506-ac57824bc0c4" (UID: "4eddddf5-9ee1-4d43-a506-ac57824bc0c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:54.178559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.178519    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-model-cache" pod "e93d0d4e-ec94-4653-9bfb-7dd27581502a" (UID: "e93d0d4e-ec94-4653-9bfb-7dd27581502a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:54.179297 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.178854    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tmp-dir" pod "e93d0d4e-ec94-4653-9bfb-7dd27581502a" (UID: "e93d0d4e-ec94-4653-9bfb-7dd27581502a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:54.179655 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.179612    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tls-certs" pod "4eddddf5-9ee1-4d43-a506-ac57824bc0c4" (UID: "4eddddf5-9ee1-4d43-a506-ac57824bc0c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:56:54.180470 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.180442    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93d0d4e-ec94-4653-9bfb-7dd27581502a-kube-api-access-z7nd6" pod "e93d0d4e-ec94-4653-9bfb-7dd27581502a" (UID: "e93d0d4e-ec94-4653-9bfb-7dd27581502a"). InnerVolumeSpecName "kube-api-access-z7nd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:54.181569 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.181537    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tls-certs" pod "e93d0d4e-ec94-4653-9bfb-7dd27581502a" (UID: "e93d0d4e-ec94-4653-9bfb-7dd27581502a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:56:54.182920 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.182888    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-dshm" pod "e93d0d4e-ec94-4653-9bfb-7dd27581502a" (UID: "e93d0d4e-ec94-4653-9bfb-7dd27581502a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:56:54.184237 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.184205    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-kube-api-access-47x8n" pod "4eddddf5-9ee1-4d43-a506-ac57824bc0c4" (UID: "4eddddf5-9ee1-4d43-a506-ac57824bc0c4"). InnerVolumeSpecName "kube-api-access-47x8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:54.276495 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.276408    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57d987d0-d800-436e-b59e-779b730a976e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d987d0-d800-436e-b59e-779b730a976e-maas-api-xks-ca\") pod \"57d987d0-d800-436e-b59e-779b730a976e\" (UID: \"57d987d0-d800-436e-b59e-779b730a976e\") "
Oct 31 12:56:54.276640 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.276561    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57d987d0-d800-436e-b59e-779b730a976e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d987d0-d800-436e-b59e-779b730a976e-maas-api-openshift-ca\") pod \"57d987d0-d800-436e-b59e-779b730a976e\" (UID: \"57d987d0-d800-436e-b59e-779b730a976e\") "
Oct 31 12:56:54.276709 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.276696    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d987d0-d800-436e-b59e-779b730a976e-kube-api-access-z6vzc\" (UniqueName: \"kubernetes.io/projected/57d987d0-d800-436e-b59e-779b730a976e-kube-api-access-z6vzc\") pod \"57d987d0-d800-436e-b59e-779b730a976e\" (UID: \"57d987d0-d800-436e-b59e-779b730a976e\") "
Oct 31 12:56:54.276941 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.276915    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d987d0-d800-436e-b59e-779b730a976e-maas-api-openshift-ca" pod "57d987d0-d800-436e-b59e-779b730a976e" (UID: "57d987d0-d800-436e-b59e-779b730a976e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:56:54.277186 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277169    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.277251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277188    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.277251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277217    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.277251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277229    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.277251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277242    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57d987d0-d800-436e-b59e-779b730a976e-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.277496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277256    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.277496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277264    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47x8n\" (UniqueName: \"kubernetes.io/projected/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-kube-api-access-47x8n\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.277496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277273    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.277496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277281    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.277496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277289    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eddddf5-9ee1-4d43-a506-ac57824bc0c4-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.277496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277296    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93d0d4e-ec94-4653-9bfb-7dd27581502a-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.277496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277304    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7nd6\" (UniqueName: \"kubernetes.io/projected/e93d0d4e-ec94-4653-9bfb-7dd27581502a-kube-api-access-z7nd6\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.277496 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.277315    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93d0d4e-ec94-4653-9bfb-7dd27581502a-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.278615 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.278593    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d987d0-d800-436e-b59e-779b730a976e-kube-api-access-z6vzc" pod "57d987d0-d800-436e-b59e-779b730a976e" (UID: "57d987d0-d800-436e-b59e-779b730a976e"). InnerVolumeSpecName "kube-api-access-z6vzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:56:54.278710 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.278636    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d987d0-d800-436e-b59e-779b730a976e-maas-api-xks-ca" pod "57d987d0-d800-436e-b59e-779b730a976e" (UID: "57d987d0-d800-436e-b59e-779b730a976e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:56:54.378225 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.378182    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6vzc\" (UniqueName: \"kubernetes.io/projected/57d987d0-d800-436e-b59e-779b730a976e-kube-api-access-z6vzc\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.378225 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.378222    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57d987d0-d800-436e-b59e-779b730a976e-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:56:54.935292 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.935259    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"
Oct 31 12:56:54.935845 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.935259    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm" event={"ID":"e93d0d4e-ec94-4653-9bfb-7dd27581502a","Type":"ContainerDied","Data":"530c3f2c90794b7e8ab6a10a3f38bba2d208870a39afb7f24176b79b9ccafc32"}
Oct 31 12:56:54.935845 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.935411    5663 scope.go:122] "RemoveContainer" containerID="95d49f43d2b23fd6143ff1577ebb53575486efb1961e8ebe999bef55e6e5055d"
Oct 31 12:56:54.936844 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.936828    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"
Oct 31 12:56:54.936930 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.936868    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2" event={"ID":"4eddddf5-9ee1-4d43-a506-ac57824bc0c4","Type":"ContainerDied","Data":"90d69d906eaa99231b50293e7cc1f67ac5a62e79851aab6210591b85e3740f04"}
Oct 31 12:56:54.938417 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.938395    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"
Oct 31 12:56:54.938516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.938409    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb" event={"ID":"57d987d0-d800-436e-b59e-779b730a976e","Type":"ContainerDied","Data":"63638a891d9ff543191ffcb9d5e37ace8e297c84dcd15709417988b3fc6d5017"}
Oct 31 12:56:54.938516 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.938467    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63638a891d9ff543191ffcb9d5e37ace8e297c84dcd15709417988b3fc6d5017"
Oct 31 12:56:54.946589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.946568    5663 scope.go:122] "RemoveContainer" containerID="79e58cb93e2407e0ac9f2c853560649c88d9c89f96f9303951dace223049a109"
Oct 31 12:56:54.964834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.964807    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"]
Oct 31 12:56:54.970972 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.970951    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/premium-w0-fb0343-kserve-5cd7b64c58-vg5rm"]
Oct 31 12:56:54.983915 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.983893    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"]
Oct 31 12:56:54.989002 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:54.988938    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fb0343/distinct-2-w0-fb0343-kserve-58487fb754-sfbw2"]
Oct 31 12:56:55.439591 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:55.439561    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" path="/var/lib/kubelet/pods/4eddddf5-9ee1-4d43-a506-ac57824bc0c4/volumes"
Oct 31 12:56:55.440036 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:55.440022    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" path="/var/lib/kubelet/pods/e93d0d4e-ec94-4653-9bfb-7dd27581502a/volumes"
Oct 31 12:56:59.036860 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:56:59.036827    5663 scope.go:122] "RemoveContainer" containerID="ed5f71d3b2bc6deb51b980bdf7c6454fd560f9773e5b9d1447c5ef06f6e995d0"
Oct 31 12:57:06.008532 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.008499    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"]
Oct 31 12:57:06.035122 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.035048    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9" podUID="096ba82b-2c3c-439c-8010-b876ea2319a8" containerName="maas-api" containerID="cri-o://ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db" gracePeriod=30
Oct 31 12:57:06.345376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.331813    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"]
Oct 31 12:57:06.384375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.380318    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-fb0343-lgnrx"]
Oct 31 12:57:06.643600 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.643571    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:57:06.793463 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.793433    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-metrics-certs\") pod \"096ba82b-2c3c-439c-8010-b876ea2319a8\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") "
Oct 31 12:57:06.793653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.793538    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096ba82b-2c3c-439c-8010-b876ea2319a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096ba82b-2c3c-439c-8010-b876ea2319a8-openshift-service-ca\") pod \"096ba82b-2c3c-439c-8010-b876ea2319a8\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") "
Oct 31 12:57:06.793653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.793627    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-maas-api-tls\") pod \"096ba82b-2c3c-439c-8010-b876ea2319a8\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") "
Oct 31 12:57:06.793746 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.793657    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096ba82b-2c3c-439c-8010-b876ea2319a8-kube-api-access-szzfh\" (UniqueName: \"kubernetes.io/projected/096ba82b-2c3c-439c-8010-b876ea2319a8-kube-api-access-szzfh\") pod \"096ba82b-2c3c-439c-8010-b876ea2319a8\" (UID: \"096ba82b-2c3c-439c-8010-b876ea2319a8\") "
Oct 31 12:57:06.793908 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.793885    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096ba82b-2c3c-439c-8010-b876ea2319a8-openshift-service-ca" pod "096ba82b-2c3c-439c-8010-b876ea2319a8" (UID: "096ba82b-2c3c-439c-8010-b876ea2319a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:57:06.794244 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.794216    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096ba82b-2c3c-439c-8010-b876ea2319a8-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:06.795779 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.795750    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096ba82b-2c3c-439c-8010-b876ea2319a8-kube-api-access-szzfh" pod "096ba82b-2c3c-439c-8010-b876ea2319a8" (UID: "096ba82b-2c3c-439c-8010-b876ea2319a8"). InnerVolumeSpecName "kube-api-access-szzfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:57:06.795779 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.795773    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-maas-api-tls" pod "096ba82b-2c3c-439c-8010-b876ea2319a8" (UID: "096ba82b-2c3c-439c-8010-b876ea2319a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:57:06.795921 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.795759    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-metrics-certs" pod "096ba82b-2c3c-439c-8010-b876ea2319a8" (UID: "096ba82b-2c3c-439c-8010-b876ea2319a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:57:06.895332 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.895247    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:06.895332 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.895279    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096ba82b-2c3c-439c-8010-b876ea2319a8-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:06.895332 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:06.895292    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szzfh\" (UniqueName: \"kubernetes.io/projected/096ba82b-2c3c-439c-8010-b876ea2319a8-kube-api-access-szzfh\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:07.043405 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:07.043332    5663 generic.go:356] "Generic (PLEG): container finished" podID="096ba82b-2c3c-439c-8010-b876ea2319a8" containerID="ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db" exitCode=0
Oct 31 12:57:07.043405 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:07.043397    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"
Oct 31 12:57:07.043869 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:07.043426    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9" event={"ID":"096ba82b-2c3c-439c-8010-b876ea2319a8","Type":"ContainerDied","Data":"ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db"}
Oct 31 12:57:07.043869 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:07.043490    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9" event={"ID":"096ba82b-2c3c-439c-8010-b876ea2319a8","Type":"ContainerDied","Data":"ba219a1000fc6de7a5ae74be7e024e446c4fe37b39ba9dd3cc5017b905b6397f"}
Oct 31 12:57:07.043869 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:07.043528    5663 scope.go:122] "RemoveContainer" containerID="ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db"
Oct 31 12:57:07.053995 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:07.053971    5663 scope.go:122] "RemoveContainer" containerID="ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db"
Oct 31 12:57:07.054307 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:57:07.054285    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db\": container with ID starting with ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db not found: ID does not exist" containerID="ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db"
Oct 31 12:57:07.054405 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:07.054317    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db"} err="failed to get container status \"ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db\": rpc error: code = NotFound desc = could not find container \"ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db\": container with ID starting with ccf9ce1d87e495c327a7289bcb11cdbd38a2bd3a8ff9f804992fc63110a993db not found: ID does not exist"
Oct 31 12:57:07.067188 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:07.067162    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"]
Oct 31 12:57:07.071952 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:07.071932    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fb0343-775d979457-qr6z9"]
Oct 31 12:57:07.439371 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:07.439327    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096ba82b-2c3c-439c-8010-b876ea2319a8" path="/var/lib/kubelet/pods/096ba82b-2c3c-439c-8010-b876ea2319a8/volumes"
Oct 31 12:57:07.439819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:07.439805    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="180ad758-df17-403d-9112-aeb3063f288a" path="/var/lib/kubelet/pods/180ad758-df17-403d-9112-aeb3063f288a/volumes"
Oct 31 12:57:15.749021 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:15.748987    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"]
Oct 31 12:57:15.750730 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:15.750671    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerName="istio-proxy" containerID="cri-o://32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a" gracePeriod=30
Oct 31 12:57:15.756522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:15.756494    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"]
Oct 31 12:57:15.756777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:15.756745    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms" podUID="ef2c3768-6d11-40a9-85a9-92aca25c3e6c" containerName="istio-proxy" containerID="cri-o://66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419" gracePeriod=30
Oct 31 12:57:21.030720 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.030658    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:57:21.077813 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.077788    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:57:21.099774 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.099731    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-kube-api-access-btv2j\" (UniqueName: \"kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-kube-api-access-btv2j\") pod \"a849ea26-cba4-4899-87a0-9eb688e0cfae\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") "
Oct 31 12:57:21.100014 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.099840    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a849ea26-cba4-4899-87a0-9eb688e0cfae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a849ea26-cba4-4899-87a0-9eb688e0cfae-istiod-ca-cert\") pod \"a849ea26-cba4-4899-87a0-9eb688e0cfae\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") "
Oct 31 12:57:21.100014 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.099907    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-certs\") pod \"a849ea26-cba4-4899-87a0-9eb688e0cfae\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") "
Oct 31 12:57:21.100014 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.099965    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-credential-socket\") pod \"a849ea26-cba4-4899-87a0-9eb688e0cfae\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") "
Oct 31 12:57:21.100312 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.100174    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-data\") pod \"a849ea26-cba4-4899-87a0-9eb688e0cfae\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") "
Oct 31 12:57:21.100312 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.100228    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-envoy\") pod \"a849ea26-cba4-4899-87a0-9eb688e0cfae\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") "
Oct 31 12:57:21.100312 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.100260    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-token\" (UniqueName: \"kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-token\") pod \"a849ea26-cba4-4899-87a0-9eb688e0cfae\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") "
Oct 31 12:57:21.100312 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.100297    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-podinfo\") pod \"a849ea26-cba4-4899-87a0-9eb688e0cfae\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") "
Oct 31 12:57:21.100828 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.100395    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-socket\") pod \"a849ea26-cba4-4899-87a0-9eb688e0cfae\" (UID: \"a849ea26-cba4-4899-87a0-9eb688e0cfae\") "
Oct 31 12:57:21.101864 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.101499    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-socket" pod "a849ea26-cba4-4899-87a0-9eb688e0cfae" (UID: "a849ea26-cba4-4899-87a0-9eb688e0cfae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.103150 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.103026    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-data" pod "a849ea26-cba4-4899-87a0-9eb688e0cfae" (UID: "a849ea26-cba4-4899-87a0-9eb688e0cfae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.103546 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.103521    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a849ea26-cba4-4899-87a0-9eb688e0cfae-istiod-ca-cert" pod "a849ea26-cba4-4899-87a0-9eb688e0cfae" (UID: "a849ea26-cba4-4899-87a0-9eb688e0cfae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:57:21.103895 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.103830    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-certs" pod "a849ea26-cba4-4899-87a0-9eb688e0cfae" (UID: "a849ea26-cba4-4899-87a0-9eb688e0cfae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.104088 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.104068    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-credential-socket" pod "a849ea26-cba4-4899-87a0-9eb688e0cfae" (UID: "a849ea26-cba4-4899-87a0-9eb688e0cfae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.115653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.115620    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-token" pod "a849ea26-cba4-4899-87a0-9eb688e0cfae" (UID: "a849ea26-cba4-4899-87a0-9eb688e0cfae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:57:21.120056 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.119969    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-podinfo" pod "a849ea26-cba4-4899-87a0-9eb688e0cfae" (UID: "a849ea26-cba4-4899-87a0-9eb688e0cfae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:57:21.120056 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.120016    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-kube-api-access-btv2j" pod "a849ea26-cba4-4899-87a0-9eb688e0cfae" (UID: "a849ea26-cba4-4899-87a0-9eb688e0cfae"). InnerVolumeSpecName "kube-api-access-btv2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:57:21.131902 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.131864    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-envoy" pod "a849ea26-cba4-4899-87a0-9eb688e0cfae" (UID: "a849ea26-cba4-4899-87a0-9eb688e0cfae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.201873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.201837    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-kube-api-access-f9h4z\" (UniqueName: \"kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-kube-api-access-f9h4z\") pod \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") "
Oct 31 12:57:21.202088 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.201901    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-token\" (UniqueName: \"kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-token\") pod \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") "
Oct 31 12:57:21.202088 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.201946    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-certs\") pod \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") "
Oct 31 12:57:21.202088 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.201982    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-credential-socket\") pod \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") "
Oct 31 12:57:21.202088 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.202054    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istiod-ca-cert\") pod \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") "
Oct 31 12:57:21.202304 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.202114    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-envoy\") pod \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") "
Oct 31 12:57:21.202304 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.202200    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-podinfo\") pod \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") "
Oct 31 12:57:21.202304 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.202278    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-certs" pod "ef2c3768-6d11-40a9-85a9-92aca25c3e6c" (UID: "ef2c3768-6d11-40a9-85a9-92aca25c3e6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.202304 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.202297    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-socket\") pod \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") "
Oct 31 12:57:21.202589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.202394    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-data\") pod \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\" (UID: \"ef2c3768-6d11-40a9-85a9-92aca25c3e6c\") "
Oct 31 12:57:21.202589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.202477    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-credential-socket" pod "ef2c3768-6d11-40a9-85a9-92aca25c3e6c" (UID: "ef2c3768-6d11-40a9-85a9-92aca25c3e6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.202705 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.202615    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istiod-ca-cert" pod "ef2c3768-6d11-40a9-85a9-92aca25c3e6c" (UID: "ef2c3768-6d11-40a9-85a9-92aca25c3e6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:57:21.203102 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.202859    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-socket" pod "ef2c3768-6d11-40a9-85a9-92aca25c3e6c" (UID: "ef2c3768-6d11-40a9-85a9-92aca25c3e6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.203102 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203065    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-data" pod "ef2c3768-6d11-40a9-85a9-92aca25c3e6c" (UID: "ef2c3768-6d11-40a9-85a9-92aca25c3e6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.203102 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203086    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.203102 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203104    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.203383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203118    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.203383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203130    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a849ea26-cba4-4899-87a0-9eb688e0cfae-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.203383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203144    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.203383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203158    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btv2j\" (UniqueName: \"kubernetes.io/projected/a849ea26-cba4-4899-87a0-9eb688e0cfae-kube-api-access-btv2j\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.203383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203173    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.203383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203187    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.203383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203200    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a849ea26-cba4-4899-87a0-9eb688e0cfae-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.203383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203212    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.203383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203225    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.203383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203238    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a849ea26-cba4-4899-87a0-9eb688e0cfae-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.203383 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.203251    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.205091 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.205072    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-token" pod "ef2c3768-6d11-40a9-85a9-92aca25c3e6c" (UID: "ef2c3768-6d11-40a9-85a9-92aca25c3e6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:57:21.205283 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.205266    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-envoy" pod "ef2c3768-6d11-40a9-85a9-92aca25c3e6c" (UID: "ef2c3768-6d11-40a9-85a9-92aca25c3e6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:57:21.206873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.206854    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-podinfo" pod "ef2c3768-6d11-40a9-85a9-92aca25c3e6c" (UID: "ef2c3768-6d11-40a9-85a9-92aca25c3e6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:57:21.210973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.210952    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-kube-api-access-f9h4z" pod "ef2c3768-6d11-40a9-85a9-92aca25c3e6c" (UID: "ef2c3768-6d11-40a9-85a9-92aca25c3e6c"). InnerVolumeSpecName "kube-api-access-f9h4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:57:21.238925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.238894    5663 generic.go:356] "Generic (PLEG): container finished" podID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerID="32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a" exitCode=0
Oct 31 12:57:21.239077 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.238964    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"
Oct 31 12:57:21.239077 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.238979    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" event={"ID":"a849ea26-cba4-4899-87a0-9eb688e0cfae","Type":"ContainerDied","Data":"32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a"}
Oct 31 12:57:21.239202 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.239074    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s" event={"ID":"a849ea26-cba4-4899-87a0-9eb688e0cfae","Type":"ContainerDied","Data":"e5c0677656f437fa1cfebff27753805dd985e9ab76bacac3f0dc55bb9847bef4"}
Oct 31 12:57:21.239202 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.239092    5663 scope.go:122] "RemoveContainer" containerID="32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a"
Oct 31 12:57:21.245935 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.245904    5663 generic.go:356] "Generic (PLEG): container finished" podID="ef2c3768-6d11-40a9-85a9-92aca25c3e6c" containerID="66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419" exitCode=0
Oct 31 12:57:21.246048 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.245951    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms" event={"ID":"ef2c3768-6d11-40a9-85a9-92aca25c3e6c","Type":"ContainerDied","Data":"66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419"}
Oct 31 12:57:21.246048 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.245998    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms" event={"ID":"ef2c3768-6d11-40a9-85a9-92aca25c3e6c","Type":"ContainerDied","Data":"e1de06b80e2fd11b9c3b04efdd9b48329ce62e6d5376f3c3b1012d4d416ef4dd"}
Oct 31 12:57:21.246143 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.246071    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"
Oct 31 12:57:21.251268 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.251248    5663 scope.go:122] "RemoveContainer" containerID="32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a"
Oct 31 12:57:21.251578 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:57:21.251554    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a\": container with ID starting with 32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a not found: ID does not exist" containerID="32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a"
Oct 31 12:57:21.251661 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.251586    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a"} err="failed to get container status \"32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a\": rpc error: code = NotFound desc = could not find container \"32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a\": container with ID starting with 32de5b516dacf7c7fccdb241aee6d138cb0a8a6b1fa7f3cf3ffe69499a791e4a not found: ID does not exist"
Oct 31 12:57:21.251661 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.251605    5663 scope.go:122] "RemoveContainer" containerID="66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419"
Oct 31 12:57:21.262281 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.262262    5663 scope.go:122] "RemoveContainer" containerID="66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419"
Oct 31 12:57:21.262634 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:57:21.262612    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419\": container with ID starting with 66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419 not found: ID does not exist" containerID="66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419"
Oct 31 12:57:21.262702 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.262640    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419"} err="failed to get container status \"66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419\": rpc error: code = NotFound desc = could not find container \"66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419\": container with ID starting with 66281d73a298a86ce1e5ebfe581495d4b594928ae821f4889ce50ed25c1ae419 not found: ID does not exist"
Oct 31 12:57:21.275134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.275107    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"]
Oct 31 12:57:21.279586 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.279565    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-f5j4s"]
Oct 31 12:57:21.288119 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.288070    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"]
Oct 31 12:57:21.295113 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.295089    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fb0343-openshift-default-5b4cb66457-525ms"]
Oct 31 12:57:21.304532 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.304511    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9h4z\" (UniqueName: \"kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-kube-api-access-f9h4z\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.304532 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.304532    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.304661 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.304544    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.304661 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.304553    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.304661 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.304563    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2c3768-6d11-40a9-85a9-92aca25c3e6c-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:57:21.440778 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.440746    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" path="/var/lib/kubelet/pods/a849ea26-cba4-4899-87a0-9eb688e0cfae/volumes"
Oct 31 12:57:21.441232 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:57:21.441219    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2c3768-6d11-40a9-85a9-92aca25c3e6c" path="/var/lib/kubelet/pods/ef2c3768-6d11-40a9-85a9-92aca25c3e6c/volumes"
Oct 31 12:57:22.476467 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:57:22.476428    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:57:52.478573 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:57:52.478539    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:58:21.741945 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.741905    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"]
Oct 31 12:58:21.743388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743345    5663 cpu_manager.go:414] "containerMap: removing container" podUID="57d987d0-d800-436e-b59e-779b730a976e" containerName="revoke-keys"
Oct 31 12:58:21.743559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743536    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d987d0-d800-436e-b59e-779b730a976e" containerName="revoke-keys"
Oct 31 12:58:21.743713 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743571    5663 cpu_manager.go:414] "containerMap: removing container" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main"
Oct 31 12:58:21.743713 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743580    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main"
Oct 31 12:58:21.743713 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743591    5663 cpu_manager.go:414] "containerMap: removing container" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerName="istio-proxy"
Oct 31 12:58:21.743713 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743598    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerName="istio-proxy"
Oct 31 12:58:21.743713 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743633    5663 cpu_manager.go:414] "containerMap: removing container" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main"
Oct 31 12:58:21.743713 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743641    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main"
Oct 31 12:58:21.743713 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743674    5663 cpu_manager.go:414] "containerMap: removing container" podUID="096ba82b-2c3c-439c-8010-b876ea2319a8" containerName="maas-api"
Oct 31 12:58:21.743713 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743681    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096ba82b-2c3c-439c-8010-b876ea2319a8" containerName="maas-api"
Oct 31 12:58:21.743713 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743705    5663 cpu_manager.go:414] "containerMap: removing container" podUID="ef2c3768-6d11-40a9-85a9-92aca25c3e6c" containerName="istio-proxy"
Oct 31 12:58:21.743713 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743712    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2c3768-6d11-40a9-85a9-92aca25c3e6c" containerName="istio-proxy"
Oct 31 12:58:21.744249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743844    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2c3768-6d11-40a9-85a9-92aca25c3e6c" containerName="istio-proxy"
Oct 31 12:58:21.744249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743860    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d987d0-d800-436e-b59e-779b730a976e" containerName="revoke-keys"
Oct 31 12:58:21.744249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743873    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93d0d4e-ec94-4653-9bfb-7dd27581502a" containerName="main"
Oct 31 12:58:21.744249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743882    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="a849ea26-cba4-4899-87a0-9eb688e0cfae" containerName="istio-proxy"
Oct 31 12:58:21.744249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743893    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eddddf5-9ee1-4d43-a506-ac57824bc0c4" containerName="main"
Oct 31 12:58:21.744249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.743905    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="096ba82b-2c3c-439c-8010-b876ea2319a8" containerName="maas-api"
Oct 31 12:58:21.748675 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.748572    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:21.749835 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.749194    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"]
Oct 31 12:58:21.754986 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.754966    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:58:21.755149 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.754979    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:58:21.929144 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.929111    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69b0a28-86pgd\" (UID: \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:21.929348 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.929170    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69b0a28-86pgd\" (UID: \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:21.929466 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:21.929401    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf55x\" (UniqueName: \"kubernetes.io/projected/21e72666-f930-4fc9-a5b7-7d42f56b9d83-kube-api-access-sf55x\") pod \"maas-api-revoke-sub-c69b0a28-86pgd\" (UID: \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:22.030224 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.030140    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf55x\" (UniqueName: \"kubernetes.io/projected/21e72666-f930-4fc9-a5b7-7d42f56b9d83-kube-api-access-sf55x\") pod \"maas-api-revoke-sub-c69b0a28-86pgd\" (UID: \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:22.030419 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.030290    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69b0a28-86pgd\" (UID: \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:22.030419 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.030338    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69b0a28-86pgd\" (UID: \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:22.031119 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.031088    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69b0a28-86pgd\" (UID: \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:22.032815 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.032795    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69b0a28-86pgd\" (UID: \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:22.042852 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.042816    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf55x\" (UniqueName: \"kubernetes.io/projected/21e72666-f930-4fc9-a5b7-7d42f56b9d83-kube-api-access-sf55x\") pod \"maas-api-revoke-sub-c69b0a28-86pgd\" (UID: \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:22.068127 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.068094    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:22.154559 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.154055    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"]
Oct 31 12:58:22.160515 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.160488    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"]
Oct 31 12:58:22.160684 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.160583    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:22.235149 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.235096    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-170f8916-qmw92\" (UID: \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:22.235149 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.235150    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-170f8916-qmw92\" (UID: \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:22.235434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.235258    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sskns\" (UniqueName: \"kubernetes.io/projected/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-kube-api-access-sskns\") pod \"maas-api-revoke-sub-170f8916-qmw92\" (UID: \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:22.336438 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.336404    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-170f8916-qmw92\" (UID: \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:22.336639 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.336449    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-170f8916-qmw92\" (UID: \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:22.336639 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.336535    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sskns\" (UniqueName: \"kubernetes.io/projected/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-kube-api-access-sskns\") pod \"maas-api-revoke-sub-170f8916-qmw92\" (UID: \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:22.337103 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.337076    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-170f8916-qmw92\" (UID: \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:22.338912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.338890    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-170f8916-qmw92\" (UID: \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:22.347320 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:58:22.347288    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e72666_f930_4fc9_a5b7_7d42f56b9d83.slice/crio-aa6307d5975ee7fc7fc69014f85436e21f79c219992c0ee7391a04c50d0a3e5d WatchSource:0}: Error finding container aa6307d5975ee7fc7fc69014f85436e21f79c219992c0ee7391a04c50d0a3e5d: Status 404 returned error can't find the container with id aa6307d5975ee7fc7fc69014f85436e21f79c219992c0ee7391a04c50d0a3e5d
Oct 31 12:58:22.347843 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.347820    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"]
Oct 31 12:58:22.350525 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.350508    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sskns\" (UniqueName: \"kubernetes.io/projected/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-kube-api-access-sskns\") pod \"maas-api-revoke-sub-170f8916-qmw92\" (UID: \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:22.476084 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:58:22.476051    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:58:22.483872 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.483845    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:22.755518 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.755480    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd" event={"ID":"21e72666-f930-4fc9-a5b7-7d42f56b9d83","Type":"ContainerStarted","Data":"0864ba6e3b1e9560548f088f94b613894c691df4f453d0bc38e0d22862124b83"}
Oct 31 12:58:22.755905 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.755540    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd" event={"ID":"21e72666-f930-4fc9-a5b7-7d42f56b9d83","Type":"ContainerStarted","Data":"aa6307d5975ee7fc7fc69014f85436e21f79c219992c0ee7391a04c50d0a3e5d"}
Oct 31 12:58:22.775977 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.775926    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd" podStartSLOduration=1.775912194 podStartE2EDuration="1.775912194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:22.772944889 +0000 UTC m=+1827.513371104" watchObservedRunningTime="2026-10-31 12:58:22.775912194 +0000 UTC m=+1827.516338404"
Oct 31 12:58:22.789914 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:22.789890    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"]
Oct 31 12:58:22.792258 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:58:22.792234    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode597fa9f_7e3e_40b2_b841_e4a7d7bed491.slice/crio-7e8a8f2f129cf201aff7dc84d963bb51b26913b0e200fc6c072b18e75934bfd1 WatchSource:0}: Error finding container 7e8a8f2f129cf201aff7dc84d963bb51b26913b0e200fc6c072b18e75934bfd1: Status 404 returned error can't find the container with id 7e8a8f2f129cf201aff7dc84d963bb51b26913b0e200fc6c072b18e75934bfd1
Oct 31 12:58:23.763044 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:23.763006    5663 generic.go:356] "Generic (PLEG): container finished" podID="e597fa9f-7e3e-40b2-b841-e4a7d7bed491" containerID="1835b6244c5489ff9e4cde31b651b1dd12e8882d21314342da2dd459e8a65537" exitCode=0
Oct 31 12:58:23.763508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:23.763096    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92" event={"ID":"e597fa9f-7e3e-40b2-b841-e4a7d7bed491","Type":"ContainerDied","Data":"1835b6244c5489ff9e4cde31b651b1dd12e8882d21314342da2dd459e8a65537"}
Oct 31 12:58:23.763508 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:23.763153    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92" event={"ID":"e597fa9f-7e3e-40b2-b841-e4a7d7bed491","Type":"ContainerStarted","Data":"7e8a8f2f129cf201aff7dc84d963bb51b26913b0e200fc6c072b18e75934bfd1"}
Oct 31 12:58:23.764679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:23.764656    5663 generic.go:356] "Generic (PLEG): container finished" podID="21e72666-f930-4fc9-a5b7-7d42f56b9d83" containerID="0864ba6e3b1e9560548f088f94b613894c691df4f453d0bc38e0d22862124b83" exitCode=0
Oct 31 12:58:23.764786 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:23.764696    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd" event={"ID":"21e72666-f930-4fc9-a5b7-7d42f56b9d83","Type":"ContainerDied","Data":"0864ba6e3b1e9560548f088f94b613894c691df4f453d0bc38e0d22862124b83"}
Oct 31 12:58:24.948283 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:24.948254    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:24.977130 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:24.977098    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-openshift-ca\") pod \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\" (UID: \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\") "
Oct 31 12:58:24.977283 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:24.977198    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-kube-api-access-sskns\" (UniqueName: \"kubernetes.io/projected/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-kube-api-access-sskns\") pod \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\" (UID: \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\") "
Oct 31 12:58:24.977283 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:24.977266    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-xks-ca\") pod \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\" (UID: \"e597fa9f-7e3e-40b2-b841-e4a7d7bed491\") "
Oct 31 12:58:24.977562 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:24.977532    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-openshift-ca" pod "e597fa9f-7e3e-40b2-b841-e4a7d7bed491" (UID: "e597fa9f-7e3e-40b2-b841-e4a7d7bed491"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:58:24.977869 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:24.977833    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:24.980097 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:24.980073    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-xks-ca" pod "e597fa9f-7e3e-40b2-b841-e4a7d7bed491" (UID: "e597fa9f-7e3e-40b2-b841-e4a7d7bed491"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:24.980194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:24.980083    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-kube-api-access-sskns" pod "e597fa9f-7e3e-40b2-b841-e4a7d7bed491" (UID: "e597fa9f-7e3e-40b2-b841-e4a7d7bed491"). InnerVolumeSpecName "kube-api-access-sskns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:25.004228 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.004202    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:25.078517 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.078489    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-xks-ca\") pod \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\" (UID: \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\") "
Oct 31 12:58:25.078704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.078568    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-openshift-ca\") pod \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\" (UID: \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\") "
Oct 31 12:58:25.078779 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.078748    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e72666-f930-4fc9-a5b7-7d42f56b9d83-kube-api-access-sf55x\" (UniqueName: \"kubernetes.io/projected/21e72666-f930-4fc9-a5b7-7d42f56b9d83-kube-api-access-sf55x\") pod \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\" (UID: \"21e72666-f930-4fc9-a5b7-7d42f56b9d83\") "
Oct 31 12:58:25.078904 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.078882    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-openshift-ca" pod "21e72666-f930-4fc9-a5b7-7d42f56b9d83" (UID: "21e72666-f930-4fc9-a5b7-7d42f56b9d83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:58:25.079291 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.079269    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sskns\" (UniqueName: \"kubernetes.io/projected/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-kube-api-access-sskns\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:25.079400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.079296    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e597fa9f-7e3e-40b2-b841-e4a7d7bed491-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:25.079400 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.079310    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:25.080548 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.080528    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-xks-ca" pod "21e72666-f930-4fc9-a5b7-7d42f56b9d83" (UID: "21e72666-f930-4fc9-a5b7-7d42f56b9d83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:25.080722 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.080702    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e72666-f930-4fc9-a5b7-7d42f56b9d83-kube-api-access-sf55x" pod "21e72666-f930-4fc9-a5b7-7d42f56b9d83" (UID: "21e72666-f930-4fc9-a5b7-7d42f56b9d83"). InnerVolumeSpecName "kube-api-access-sf55x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:25.180341 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.180302    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e72666-f930-4fc9-a5b7-7d42f56b9d83-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:25.180341 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.180339    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf55x\" (UniqueName: \"kubernetes.io/projected/21e72666-f930-4fc9-a5b7-7d42f56b9d83-kube-api-access-sf55x\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:25.778925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.778836    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"
Oct 31 12:58:25.778925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.778866    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92" event={"ID":"e597fa9f-7e3e-40b2-b841-e4a7d7bed491","Type":"ContainerDied","Data":"7e8a8f2f129cf201aff7dc84d963bb51b26913b0e200fc6c072b18e75934bfd1"}
Oct 31 12:58:25.778925 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.778918    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e8a8f2f129cf201aff7dc84d963bb51b26913b0e200fc6c072b18e75934bfd1"
Oct 31 12:58:25.781086 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.781053    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd" event={"ID":"21e72666-f930-4fc9-a5b7-7d42f56b9d83","Type":"ContainerDied","Data":"aa6307d5975ee7fc7fc69014f85436e21f79c219992c0ee7391a04c50d0a3e5d"}
Oct 31 12:58:25.781251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.781094    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"
Oct 31 12:58:25.781251 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:25.781107    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa6307d5975ee7fc7fc69014f85436e21f79c219992c0ee7391a04c50d0a3e5d"
Oct 31 12:58:33.009254 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.009153    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"]
Oct 31 12:58:33.009845 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.009788    5663 cpu_manager.go:414] "containerMap: removing container" podUID="21e72666-f930-4fc9-a5b7-7d42f56b9d83" containerName="revoke-keys"
Oct 31 12:58:33.009845 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.009804    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e72666-f930-4fc9-a5b7-7d42f56b9d83" containerName="revoke-keys"
Oct 31 12:58:33.009845 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.009818    5663 cpu_manager.go:414] "containerMap: removing container" podUID="e597fa9f-7e3e-40b2-b841-e4a7d7bed491" containerName="revoke-keys"
Oct 31 12:58:33.009845 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.009824    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e597fa9f-7e3e-40b2-b841-e4a7d7bed491" containerName="revoke-keys"
Oct 31 12:58:33.009989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.009916    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e72666-f930-4fc9-a5b7-7d42f56b9d83" containerName="revoke-keys"
Oct 31 12:58:33.009989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.009926    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="e597fa9f-7e3e-40b2-b841-e4a7d7bed491" containerName="revoke-keys"
Oct 31 12:58:33.013869 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.013846    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:33.034378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.033401    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:58:33.034378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.033440    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:58:33.036542 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.036521    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"]
Oct 31 12:58:33.117276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.117232    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcrl8\" (UniqueName: \"kubernetes.io/projected/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-kube-api-access-pcrl8\") pod \"maas-api-revoke-sub-743d23d0-5c4vt\" (UID: \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:33.117490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.117454    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-743d23d0-5c4vt\" (UID: \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:33.117600 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.117577    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-743d23d0-5c4vt\" (UID: \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:33.219053 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.219020    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-743d23d0-5c4vt\" (UID: \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:33.219234 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.219092    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-743d23d0-5c4vt\" (UID: \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:33.219234 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.219149    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcrl8\" (UniqueName: \"kubernetes.io/projected/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-kube-api-access-pcrl8\") pod \"maas-api-revoke-sub-743d23d0-5c4vt\" (UID: \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:33.219848 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.219819    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-743d23d0-5c4vt\" (UID: \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:33.221622 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.221596    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-743d23d0-5c4vt\" (UID: \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:33.233822 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.233795    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcrl8\" (UniqueName: \"kubernetes.io/projected/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-kube-api-access-pcrl8\") pod \"maas-api-revoke-sub-743d23d0-5c4vt\" (UID: \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:33.327674 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.327635    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:33.611525 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.611498    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"]
Oct 31 12:58:33.613681 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:58:33.613653    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c66a3a3_5c0b_4c44_a3d6_f80149fc4244.slice/crio-389431fd5a0f5aa8f6a8041f3ea741c3418cc796c9c646cd639e550367200ade WatchSource:0}: Error finding container 389431fd5a0f5aa8f6a8041f3ea741c3418cc796c9c646cd639e550367200ade: Status 404 returned error can't find the container with id 389431fd5a0f5aa8f6a8041f3ea741c3418cc796c9c646cd639e550367200ade
Oct 31 12:58:33.832697 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:33.832667    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt" event={"ID":"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244","Type":"ContainerStarted","Data":"389431fd5a0f5aa8f6a8041f3ea741c3418cc796c9c646cd639e550367200ade"}
Oct 31 12:58:34.760117 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:34.759418    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"]
Oct 31 12:58:34.819777 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:34.819742    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:34.822209 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:34.822132    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"]
Oct 31 12:58:34.846472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:34.846438    5663 generic.go:356] "Generic (PLEG): container finished" podID="8c66a3a3-5c0b-4c44-a3d6-f80149fc4244" containerID="23d2ddcb8e8a1ae6cceb16b8228977f923431aae2807c18b1e524c25ebebfa68" exitCode=0
Oct 31 12:58:34.846642 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:34.846478    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt" event={"ID":"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244","Type":"ContainerDied","Data":"23d2ddcb8e8a1ae6cceb16b8228977f923431aae2807c18b1e524c25ebebfa68"}
Oct 31 12:58:34.955379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:34.946548    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"]
Oct 31 12:58:34.959424 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:34.959388    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qklh\" (UniqueName: \"kubernetes.io/projected/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-kube-api-access-2qklh\") pod \"maas-api-revoke-sub-eb4ce7a1-22wwh\" (UID: \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:34.959566 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:34.959486    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4ce7a1-22wwh\" (UID: \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:34.959632 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:34.959590    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4ce7a1-22wwh\" (UID: \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:35.073473 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:35.064339    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qklh\" (UniqueName: \"kubernetes.io/projected/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-kube-api-access-2qklh\") pod \"maas-api-revoke-sub-eb4ce7a1-22wwh\" (UID: \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:35.073473 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:35.064473    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4ce7a1-22wwh\" (UID: \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:35.073473 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:35.064566    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4ce7a1-22wwh\" (UID: \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:35.073473 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:35.065682    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4ce7a1-22wwh\" (UID: \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:35.104631 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:35.103754    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4ce7a1-22wwh\" (UID: \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:35.131378 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:35.124830    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qklh\" (UniqueName: \"kubernetes.io/projected/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-kube-api-access-2qklh\") pod \"maas-api-revoke-sub-eb4ce7a1-22wwh\" (UID: \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:35.144796 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:35.144764    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:35.452307 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:35.452280    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"]
Oct 31 12:58:35.454607 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:58:35.454579    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1abbd9_1ed5_4842_87bb_ff328bb9d7a6.slice/crio-73f0e3337d73e3927c88f2d2c23e2b213cda063fc0c4964bf5bd4fb40487d096 WatchSource:0}: Error finding container 73f0e3337d73e3927c88f2d2c23e2b213cda063fc0c4964bf5bd4fb40487d096: Status 404 returned error can't find the container with id 73f0e3337d73e3927c88f2d2c23e2b213cda063fc0c4964bf5bd4fb40487d096
Oct 31 12:58:35.853924 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:35.853891    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh" event={"ID":"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6","Type":"ContainerStarted","Data":"73f0e3337d73e3927c88f2d2c23e2b213cda063fc0c4964bf5bd4fb40487d096"}
Oct 31 12:58:36.007133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.007106    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:36.180749 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.180671    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-openshift-ca\") pod \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\" (UID: \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\") "
Oct 31 12:58:36.180749 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.180721    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-xks-ca\") pod \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\" (UID: \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\") "
Oct 31 12:58:36.180940 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.180929    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-kube-api-access-pcrl8\" (UniqueName: \"kubernetes.io/projected/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-kube-api-access-pcrl8\") pod \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\" (UID: \"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244\") "
Oct 31 12:58:36.181160 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.181133    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-openshift-ca" pod "8c66a3a3-5c0b-4c44-a3d6-f80149fc4244" (UID: "8c66a3a3-5c0b-4c44-a3d6-f80149fc4244"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:58:36.181524 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.181503    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:36.182959 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.182936    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-xks-ca" pod "8c66a3a3-5c0b-4c44-a3d6-f80149fc4244" (UID: "8c66a3a3-5c0b-4c44-a3d6-f80149fc4244"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:36.183053 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.182967    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-kube-api-access-pcrl8" pod "8c66a3a3-5c0b-4c44-a3d6-f80149fc4244" (UID: "8c66a3a3-5c0b-4c44-a3d6-f80149fc4244"). InnerVolumeSpecName "kube-api-access-pcrl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:36.282904 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.282875    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcrl8\" (UniqueName: \"kubernetes.io/projected/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-kube-api-access-pcrl8\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:36.282904 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.282903    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:36.861253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.861222    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"
Oct 31 12:58:36.861253 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.861236    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt" event={"ID":"8c66a3a3-5c0b-4c44-a3d6-f80149fc4244","Type":"ContainerDied","Data":"389431fd5a0f5aa8f6a8041f3ea741c3418cc796c9c646cd639e550367200ade"}
Oct 31 12:58:36.861755 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.861286    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="389431fd5a0f5aa8f6a8041f3ea741c3418cc796c9c646cd639e550367200ade"
Oct 31 12:58:36.868118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.868071    5663 generic.go:356] "Generic (PLEG): container finished" podID="0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6" containerID="6eeae6ba8fcbc88e0fba030fe75e2623d4f74c6472d3dd5517e923d93f8ad196" exitCode=0
Oct 31 12:58:36.868519 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:36.868494    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh" event={"ID":"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6","Type":"ContainerDied","Data":"6eeae6ba8fcbc88e0fba030fe75e2623d4f74c6472d3dd5517e923d93f8ad196"}
Oct 31 12:58:38.063503 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.063475    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:38.210200 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.210119    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-openshift-ca\") pod \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\" (UID: \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\") "
Oct 31 12:58:38.210349 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.210222    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-kube-api-access-2qklh\" (UniqueName: \"kubernetes.io/projected/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-kube-api-access-2qklh\") pod \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\" (UID: \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\") "
Oct 31 12:58:38.210426 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.210410    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-xks-ca\") pod \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\" (UID: \"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6\") "
Oct 31 12:58:38.210532 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.210511    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-openshift-ca" pod "0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6" (UID: "0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:58:38.210814 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.210795    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:38.212379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.212329    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-xks-ca" pod "0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6" (UID: "0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:38.212379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.212373    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-kube-api-access-2qklh" pod "0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6" (UID: "0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6"). InnerVolumeSpecName "kube-api-access-2qklh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:38.312249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.312211    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qklh\" (UniqueName: \"kubernetes.io/projected/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-kube-api-access-2qklh\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:38.312249 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.312244    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:38.890215 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.890186    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"
Oct 31 12:58:38.890447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.890215    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh" event={"ID":"0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6","Type":"ContainerDied","Data":"73f0e3337d73e3927c88f2d2c23e2b213cda063fc0c4964bf5bd4fb40487d096"}
Oct 31 12:58:38.890447 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:38.890264    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73f0e3337d73e3927c88f2d2c23e2b213cda063fc0c4964bf5bd4fb40487d096"
Oct 31 12:58:43.846748 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.846715    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"]
Oct 31 12:58:43.847277 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.847264    5663 cpu_manager.go:414] "containerMap: removing container" podUID="0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6" containerName="revoke-keys"
Oct 31 12:58:43.847323 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.847278    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6" containerName="revoke-keys"
Oct 31 12:58:43.847323 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.847289    5663 cpu_manager.go:414] "containerMap: removing container" podUID="8c66a3a3-5c0b-4c44-a3d6-f80149fc4244" containerName="revoke-keys"
Oct 31 12:58:43.847323 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.847294    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c66a3a3-5c0b-4c44-a3d6-f80149fc4244" containerName="revoke-keys"
Oct 31 12:58:43.847455 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.847397    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6" containerName="revoke-keys"
Oct 31 12:58:43.847455 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.847405    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c66a3a3-5c0b-4c44-a3d6-f80149fc4244" containerName="revoke-keys"
Oct 31 12:58:43.851028 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.850999    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:43.858116 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.858085    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:58:43.858277 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.858131    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:58:43.861179 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.860789    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"]
Oct 31 12:58:43.913907 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.913867    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj\" (UID: \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:43.914165 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.914138    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbvh2\" (UniqueName: \"kubernetes.io/projected/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-kube-api-access-gbvh2\") pod \"maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj\" (UID: \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:43.914263 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:43.914182    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj\" (UID: \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:44.015522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:44.015486    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbvh2\" (UniqueName: \"kubernetes.io/projected/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-kube-api-access-gbvh2\") pod \"maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj\" (UID: \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:44.015707 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:44.015545    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj\" (UID: \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:44.015707 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:44.015684    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj\" (UID: \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:44.016233 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:44.016209    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj\" (UID: \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:44.018134 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:44.018111    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj\" (UID: \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:44.030098 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:44.030070    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbvh2\" (UniqueName: \"kubernetes.io/projected/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-kube-api-access-gbvh2\") pod \"maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj\" (UID: \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:44.182061 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:44.181977    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:44.465366 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:44.465327    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"]
Oct 31 12:58:44.466430 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:58:44.466401    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8f3ff2_e2d1_41a3_8059_22907fdbabe5.slice/crio-f512f8ad8156ac04a11e81ec7f08bd7f7c1c73fe1f21c3b2c8788934779d23e3 WatchSource:0}: Error finding container f512f8ad8156ac04a11e81ec7f08bd7f7c1c73fe1f21c3b2c8788934779d23e3: Status 404 returned error can't find the container with id f512f8ad8156ac04a11e81ec7f08bd7f7c1c73fe1f21c3b2c8788934779d23e3
Oct 31 12:58:44.936934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:44.936899    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj" event={"ID":"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5","Type":"ContainerStarted","Data":"ac606627a28f5f7bf827cb0f9461e405b7b91479ecacdb37a5acb8e8195d5b6c"}
Oct 31 12:58:44.937338 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:44.936964    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj" event={"ID":"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5","Type":"ContainerStarted","Data":"f512f8ad8156ac04a11e81ec7f08bd7f7c1c73fe1f21c3b2c8788934779d23e3"}
Oct 31 12:58:44.959083 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:44.959024    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj" podStartSLOduration=1.959005131 podStartE2EDuration="1.959005131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:44.955005152 +0000 UTC m=+1849.695431369" watchObservedRunningTime="2026-10-31 12:58:44.959005131 +0000 UTC m=+1849.699431348"
Oct 31 12:58:45.813373 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:45.810495    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"]
Oct 31 12:58:46.013318 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:46.013272    5663 generic.go:356] "Generic (PLEG): container finished" podID="ab8f3ff2-e2d1-41a3-8059-22907fdbabe5" containerID="ac606627a28f5f7bf827cb0f9461e405b7b91479ecacdb37a5acb8e8195d5b6c" exitCode=0
Oct 31 12:58:46.014038 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:46.014009    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj" event={"ID":"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5","Type":"ContainerDied","Data":"ac606627a28f5f7bf827cb0f9461e405b7b91479ecacdb37a5acb8e8195d5b6c"}
Oct 31 12:58:47.187229 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:47.187197    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:47.311631 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:47.311597    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-kube-api-access-gbvh2\" (UniqueName: \"kubernetes.io/projected/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-kube-api-access-gbvh2\") pod \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\" (UID: \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\") "
Oct 31 12:58:47.311804 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:47.311659    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-openshift-ca\") pod \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\" (UID: \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\") "
Oct 31 12:58:47.311862 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:47.311842    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-xks-ca\") pod \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\" (UID: \"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5\") "
Oct 31 12:58:47.312037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:47.312012    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-openshift-ca" pod "ab8f3ff2-e2d1-41a3-8059-22907fdbabe5" (UID: "ab8f3ff2-e2d1-41a3-8059-22907fdbabe5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:58:47.312470 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:47.312444    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:47.313944 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:47.313921    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-kube-api-access-gbvh2" pod "ab8f3ff2-e2d1-41a3-8059-22907fdbabe5" (UID: "ab8f3ff2-e2d1-41a3-8059-22907fdbabe5"). InnerVolumeSpecName "kube-api-access-gbvh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:47.313944 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:47.313921    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-xks-ca" pod "ab8f3ff2-e2d1-41a3-8059-22907fdbabe5" (UID: "ab8f3ff2-e2d1-41a3-8059-22907fdbabe5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:47.414659 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:47.414572    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbvh2\" (UniqueName: \"kubernetes.io/projected/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-kube-api-access-gbvh2\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:47.414659 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:47.414624    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:48.028108 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:48.028078    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"
Oct 31 12:58:48.028300 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:48.028102    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj" event={"ID":"ab8f3ff2-e2d1-41a3-8059-22907fdbabe5","Type":"ContainerDied","Data":"f512f8ad8156ac04a11e81ec7f08bd7f7c1c73fe1f21c3b2c8788934779d23e3"}
Oct 31 12:58:48.028300 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:48.028168    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f512f8ad8156ac04a11e81ec7f08bd7f7c1c73fe1f21c3b2c8788934779d23e3"
Oct 31 12:58:49.949267 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:49.949221    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"]
Oct 31 12:58:49.985860 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:49.985791    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv" podUID="3c7e399c-2f1e-4d61-8116-68b897e31ed1" containerName="main" containerID="cri-o://bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043" gracePeriod=45
Oct 31 12:58:50.105380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.100399    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"]
Oct 31 12:58:50.105380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.101246    5663 cpu_manager.go:414] "containerMap: removing container" podUID="ab8f3ff2-e2d1-41a3-8059-22907fdbabe5" containerName="revoke-keys"
Oct 31 12:58:50.105380 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.101266    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8f3ff2-e2d1-41a3-8059-22907fdbabe5" containerName="revoke-keys"
Oct 31 12:58:50.119379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.105785    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab8f3ff2-e2d1-41a3-8059-22907fdbabe5" containerName="revoke-keys"
Oct 31 12:58:50.138054 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.134079    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:50.162394 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.140651    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:58:50.162394 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.141382    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:58:50.162394 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.158137    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"]
Oct 31 12:58:50.182388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.173779    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d176a5-db4df\" (UID: \"1da38c69-768b-4569-b55e-43baf1f1e0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:50.182388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.173838    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxmg9\" (UniqueName: \"kubernetes.io/projected/1da38c69-768b-4569-b55e-43baf1f1e0bb-kube-api-access-sxmg9\") pod \"maas-api-revoke-keys-e2e-inf-a-d176a5-db4df\" (UID: \"1da38c69-768b-4569-b55e-43baf1f1e0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:50.182388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.173886    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d176a5-db4df\" (UID: \"1da38c69-768b-4569-b55e-43baf1f1e0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:50.274763 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.274727    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d176a5-db4df\" (UID: \"1da38c69-768b-4569-b55e-43baf1f1e0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:50.274973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.274776    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxmg9\" (UniqueName: \"kubernetes.io/projected/1da38c69-768b-4569-b55e-43baf1f1e0bb-kube-api-access-sxmg9\") pod \"maas-api-revoke-keys-e2e-inf-a-d176a5-db4df\" (UID: \"1da38c69-768b-4569-b55e-43baf1f1e0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:50.274973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.274814    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d176a5-db4df\" (UID: \"1da38c69-768b-4569-b55e-43baf1f1e0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:50.275499 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.275475    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d176a5-db4df\" (UID: \"1da38c69-768b-4569-b55e-43baf1f1e0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:50.277709 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.277685    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d176a5-db4df\" (UID: \"1da38c69-768b-4569-b55e-43baf1f1e0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:50.309461 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.309254    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxmg9\" (UniqueName: \"kubernetes.io/projected/1da38c69-768b-4569-b55e-43baf1f1e0bb-kube-api-access-sxmg9\") pod \"maas-api-revoke-keys-e2e-inf-a-d176a5-db4df\" (UID: \"1da38c69-768b-4569-b55e-43baf1f1e0bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:50.388875 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.388849    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:58:50.478026 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.477930    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-dshm\") pod \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") "
Oct 31 12:58:50.478201 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.478052    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-model-cache\") pod \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") "
Oct 31 12:58:50.478201 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.478106    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tmp-dir\") pod \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") "
Oct 31 12:58:50.478201 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.478188    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tls-certs\") pod \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") "
Oct 31 12:58:50.478389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.478242    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7e399c-2f1e-4d61-8116-68b897e31ed1-kube-api-access-8kz6n\" (UniqueName: \"kubernetes.io/projected/3c7e399c-2f1e-4d61-8116-68b897e31ed1-kube-api-access-8kz6n\") pod \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") "
Oct 31 12:58:50.478389 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.478300    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-home\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-home\") pod \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\" (UID: \"3c7e399c-2f1e-4d61-8116-68b897e31ed1\") "
Oct 31 12:58:50.479130 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.479101    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-home" pod "3c7e399c-2f1e-4d61-8116-68b897e31ed1" (UID: "3c7e399c-2f1e-4d61-8116-68b897e31ed1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:58:50.479265 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.479163    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tmp-dir" pod "3c7e399c-2f1e-4d61-8116-68b897e31ed1" (UID: "3c7e399c-2f1e-4d61-8116-68b897e31ed1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:58:50.479325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.479276    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-model-cache" pod "3c7e399c-2f1e-4d61-8116-68b897e31ed1" (UID: "3c7e399c-2f1e-4d61-8116-68b897e31ed1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:58:50.480762 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.480727    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-dshm" pod "3c7e399c-2f1e-4d61-8116-68b897e31ed1" (UID: "3c7e399c-2f1e-4d61-8116-68b897e31ed1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:58:50.485554 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.485524    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tls-certs" pod "3c7e399c-2f1e-4d61-8116-68b897e31ed1" (UID: "3c7e399c-2f1e-4d61-8116-68b897e31ed1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:50.486049 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.486024    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7e399c-2f1e-4d61-8116-68b897e31ed1-kube-api-access-8kz6n" pod "3c7e399c-2f1e-4d61-8116-68b897e31ed1" (UID: "3c7e399c-2f1e-4d61-8116-68b897e31ed1"). InnerVolumeSpecName "kube-api-access-8kz6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:50.505016 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.504992    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:50.579939 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.579903    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:50.580129 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.579944    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:50.580129 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.579960    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c7e399c-2f1e-4d61-8116-68b897e31ed1-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:50.580129 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.579974    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kz6n\" (UniqueName: \"kubernetes.io/projected/3c7e399c-2f1e-4d61-8116-68b897e31ed1-kube-api-access-8kz6n\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:50.580129 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.579990    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:50.580129 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.580002    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c7e399c-2f1e-4d61-8116-68b897e31ed1-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:50.793192 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:50.793164    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"]
Oct 31 12:58:50.794516 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:58:50.794490    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da38c69_768b_4569_b55e_43baf1f1e0bb.slice/crio-679190a462ae3917fc88255866db01a5f323c44ca40b0cea44a9f7db56841476 WatchSource:0}: Error finding container 679190a462ae3917fc88255866db01a5f323c44ca40b0cea44a9f7db56841476: Status 404 returned error can't find the container with id 679190a462ae3917fc88255866db01a5f323c44ca40b0cea44a9f7db56841476
Oct 31 12:58:51.228390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.223330    5663 generic.go:356] "Generic (PLEG): container finished" podID="3c7e399c-2f1e-4d61-8116-68b897e31ed1" containerID="bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043" exitCode=0
Oct 31 12:58:51.228390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.223493    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv" event={"ID":"3c7e399c-2f1e-4d61-8116-68b897e31ed1","Type":"ContainerDied","Data":"bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043"}
Oct 31 12:58:51.228390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.223531    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv" event={"ID":"3c7e399c-2f1e-4d61-8116-68b897e31ed1","Type":"ContainerDied","Data":"1f4074580cf4a044f1061ada8b5aef2ccd98d492602f14554b1436f9a35afa78"}
Oct 31 12:58:51.228390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.223552    5663 scope.go:122] "RemoveContainer" containerID="bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043"
Oct 31 12:58:51.228390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.223700    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"
Oct 31 12:58:51.244116 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.243703    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df" event={"ID":"1da38c69-768b-4569-b55e-43baf1f1e0bb","Type":"ContainerStarted","Data":"6c2224460d4ea139199f31b25ce189cd071ddabed00023d8468d2e4fbfc51518"}
Oct 31 12:58:51.244116 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.243788    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df" event={"ID":"1da38c69-768b-4569-b55e-43baf1f1e0bb","Type":"ContainerStarted","Data":"679190a462ae3917fc88255866db01a5f323c44ca40b0cea44a9f7db56841476"}
Oct 31 12:58:51.255910 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.255882    5663 scope.go:122] "RemoveContainer" containerID="bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043"
Oct 31 12:58:51.256293 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:58:51.256263    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043\": container with ID starting with bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043 not found: ID does not exist" containerID="bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043"
Oct 31 12:58:51.256396 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.256305    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043"} err="failed to get container status \"bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043\": rpc error: code = NotFound desc = could not find container \"bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043\": container with ID starting with bd0be4fd503762fb52138a2a7fc7958ca948ce5072f9a0873acb7acbf4409043 not found: ID does not exist"
Oct 31 12:58:51.262668 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.262638    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"]
Oct 31 12:58:51.265687 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.265661    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d176a5/test-model-d176a5-kserve-78cc77dcf9-pt4vv"]
Oct 31 12:58:51.281706 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.281653    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df" podStartSLOduration=1.281636357 podStartE2EDuration="1.281636357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:58:51.279696989 +0000 UTC m=+1856.020123206" watchObservedRunningTime="2026-10-31 12:58:51.281636357 +0000 UTC m=+1856.022062572"
Oct 31 12:58:51.439429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:51.439323    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7e399c-2f1e-4d61-8116-68b897e31ed1" path="/var/lib/kubelet/pods/3c7e399c-2f1e-4d61-8116-68b897e31ed1/volumes"
Oct 31 12:58:52.253185 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:52.253152    5663 generic.go:356] "Generic (PLEG): container finished" podID="1da38c69-768b-4569-b55e-43baf1f1e0bb" containerID="6c2224460d4ea139199f31b25ce189cd071ddabed00023d8468d2e4fbfc51518" exitCode=0
Oct 31 12:58:52.253621 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:52.253196    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df" event={"ID":"1da38c69-768b-4569-b55e-43baf1f1e0bb","Type":"ContainerDied","Data":"6c2224460d4ea139199f31b25ce189cd071ddabed00023d8468d2e4fbfc51518"}
Oct 31 12:58:52.477088 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:58:52.477042    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:58:53.410562 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:53.410530    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:53.536633 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:53.536532    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da38c69-768b-4569-b55e-43baf1f1e0bb-kube-api-access-sxmg9\" (UniqueName: \"kubernetes.io/projected/1da38c69-768b-4569-b55e-43baf1f1e0bb-kube-api-access-sxmg9\") pod \"1da38c69-768b-4569-b55e-43baf1f1e0bb\" (UID: \"1da38c69-768b-4569-b55e-43baf1f1e0bb\") "
Oct 31 12:58:53.536633 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:53.536575    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-openshift-ca\") pod \"1da38c69-768b-4569-b55e-43baf1f1e0bb\" (UID: \"1da38c69-768b-4569-b55e-43baf1f1e0bb\") "
Oct 31 12:58:53.536861 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:53.536722    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-xks-ca\") pod \"1da38c69-768b-4569-b55e-43baf1f1e0bb\" (UID: \"1da38c69-768b-4569-b55e-43baf1f1e0bb\") "
Oct 31 12:58:53.537043 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:53.537003    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-openshift-ca" pod "1da38c69-768b-4569-b55e-43baf1f1e0bb" (UID: "1da38c69-768b-4569-b55e-43baf1f1e0bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:58:53.537913 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:53.537806    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:53.539459 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:53.539430    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da38c69-768b-4569-b55e-43baf1f1e0bb-kube-api-access-sxmg9" pod "1da38c69-768b-4569-b55e-43baf1f1e0bb" (UID: "1da38c69-768b-4569-b55e-43baf1f1e0bb"). InnerVolumeSpecName "kube-api-access-sxmg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:58:53.539870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:53.539837    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-xks-ca" pod "1da38c69-768b-4569-b55e-43baf1f1e0bb" (UID: "1da38c69-768b-4569-b55e-43baf1f1e0bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:58:53.663329 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:53.646242    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxmg9\" (UniqueName: \"kubernetes.io/projected/1da38c69-768b-4569-b55e-43baf1f1e0bb-kube-api-access-sxmg9\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:53.663704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:53.663678    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1da38c69-768b-4569-b55e-43baf1f1e0bb-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:58:54.018721 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:54.018681    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"]
Oct 31 12:58:54.269310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:54.269203    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df" event={"ID":"1da38c69-768b-4569-b55e-43baf1f1e0bb","Type":"ContainerDied","Data":"679190a462ae3917fc88255866db01a5f323c44ca40b0cea44a9f7db56841476"}
Oct 31 12:58:54.269535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:54.269316    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="679190a462ae3917fc88255866db01a5f323c44ca40b0cea44a9f7db56841476"
Oct 31 12:58:54.269535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:54.269443    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"
Oct 31 12:58:59.029768 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:58:59.029715    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7" podUID="71f3a982-4cac-4228-b891-dca75a989765" containerName="payload-pre-processing" containerID="cri-o://9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3" gracePeriod=25
Oct 31 12:59:00.253603 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:00.253568    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"]
Oct 31 12:59:00.851234 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:00.851195    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2" podUID="d5b51a9d-7551-43d7-85a1-8a61abd5cfc9" containerName="main" containerID="cri-o://363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f" gracePeriod=45
Oct 31 12:59:01.135060 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.135035    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:59:01.267820 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.267787    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-kube-api-access-6bszx\" (UniqueName: \"kubernetes.io/projected/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-kube-api-access-6bszx\") pod \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") "
Oct 31 12:59:01.268248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.267903    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-dshm\") pod \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") "
Oct 31 12:59:01.268248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.267987    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tmp-dir\") pod \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") "
Oct 31 12:59:01.268248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.268009    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tls-certs\") pod \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") "
Oct 31 12:59:01.268248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.268048    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-model-cache\") pod \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") "
Oct 31 12:59:01.268248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.268076    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-home\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-home\") pod \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\" (UID: \"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9\") "
Oct 31 12:59:01.268566 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.268314    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tmp-dir" pod "d5b51a9d-7551-43d7-85a1-8a61abd5cfc9" (UID: "d5b51a9d-7551-43d7-85a1-8a61abd5cfc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:01.268566 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.268454    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-model-cache" pod "d5b51a9d-7551-43d7-85a1-8a61abd5cfc9" (UID: "d5b51a9d-7551-43d7-85a1-8a61abd5cfc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:01.268566 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.268466    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-home" pod "d5b51a9d-7551-43d7-85a1-8a61abd5cfc9" (UID: "d5b51a9d-7551-43d7-85a1-8a61abd5cfc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:01.268772 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.268753    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:01.268826 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.268781    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:01.268826 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.268799    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:01.270214 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.270185    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-dshm" pod "d5b51a9d-7551-43d7-85a1-8a61abd5cfc9" (UID: "d5b51a9d-7551-43d7-85a1-8a61abd5cfc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:01.270323 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.270282    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-kube-api-access-6bszx" pod "d5b51a9d-7551-43d7-85a1-8a61abd5cfc9" (UID: "d5b51a9d-7551-43d7-85a1-8a61abd5cfc9"). InnerVolumeSpecName "kube-api-access-6bszx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:01.270323 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.270299    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tls-certs" pod "d5b51a9d-7551-43d7-85a1-8a61abd5cfc9" (UID: "d5b51a9d-7551-43d7-85a1-8a61abd5cfc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:01.332562 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.332526    5663 generic.go:356] "Generic (PLEG): container finished" podID="d5b51a9d-7551-43d7-85a1-8a61abd5cfc9" containerID="363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f" exitCode=0
Oct 31 12:59:01.332720 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.332576    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"
Oct 31 12:59:01.332720 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.332607    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2" event={"ID":"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9","Type":"ContainerDied","Data":"363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f"}
Oct 31 12:59:01.332720 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.332652    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2" event={"ID":"d5b51a9d-7551-43d7-85a1-8a61abd5cfc9","Type":"ContainerDied","Data":"ca91400007be504c0317d037df49aaca72f1c6cdd4e5747cd51a97890c7e7ede"}
Oct 31 12:59:01.332720 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.332669    5663 scope.go:122] "RemoveContainer" containerID="363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f"
Oct 31 12:59:01.343499 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.343480    5663 scope.go:122] "RemoveContainer" containerID="363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f"
Oct 31 12:59:01.343784 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:01.343763    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f\": container with ID starting with 363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f not found: ID does not exist" containerID="363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f"
Oct 31 12:59:01.343824 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.343793    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f"} err="failed to get container status \"363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f\": rpc error: code = NotFound desc = could not find container \"363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f\": container with ID starting with 363ee25433cd213fc5e50462e3601ac8e61906625e149cb2bbfe1e551f91591f not found: ID does not exist"
Oct 31 12:59:01.357173 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.357147    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"]
Oct 31 12:59:01.361895 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.361848    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2c393f/test-model-2c393f-kserve-548cc44567-q66p2"]
Oct 31 12:59:01.369770 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.369749    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bszx\" (UniqueName: \"kubernetes.io/projected/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-kube-api-access-6bszx\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:01.369839 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.369772    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:01.369839 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.369783    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:01.439602 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.439574    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b51a9d-7551-43d7-85a1-8a61abd5cfc9" path="/var/lib/kubelet/pods/d5b51a9d-7551-43d7-85a1-8a61abd5cfc9/volumes"
Oct 31 12:59:01.913902 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.913867    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"]
Oct 31 12:59:01.923941 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:01.923908    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d086f3-p4gnj"]
Oct 31 12:59:03.439865 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:03.439824    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab8f3ff2-e2d1-41a3-8059-22907fdbabe5" path="/var/lib/kubelet/pods/ab8f3ff2-e2d1-41a3-8059-22907fdbabe5/volumes"
Oct 31 12:59:04.978999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:04.978957    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64" podUID="d7343f75-22c7-4f7c-9e73-8edcdb5a18e4" containerName="main" containerID="cri-o://88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45" gracePeriod=45
Oct 31 12:59:05.242330 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.242268    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:59:05.254002 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.253969    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj" podUID="1ceb5e4b-0314-4f7e-bd59-ec883e6db112" containerName="payload-processing" containerID="cri-o://2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92" gracePeriod=25
Oct 31 12:59:05.332794 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.332760    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-dshm\") pod \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") "
Oct 31 12:59:05.332989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.332821    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-home\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-home\") pod \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") "
Oct 31 12:59:05.332989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.332895    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-kube-api-access-brllg\" (UniqueName: \"kubernetes.io/projected/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-kube-api-access-brllg\") pod \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") "
Oct 31 12:59:05.332989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.332942    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tls-certs\") pod \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") "
Oct 31 12:59:05.333236 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.333051    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-model-cache\") pod \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") "
Oct 31 12:59:05.333236 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.333076    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tmp-dir\") pod \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\" (UID: \"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4\") "
Oct 31 12:59:05.333236 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.333125    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-home" pod "d7343f75-22c7-4f7c-9e73-8edcdb5a18e4" (UID: "d7343f75-22c7-4f7c-9e73-8edcdb5a18e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:05.333443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.333311    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-model-cache" pod "d7343f75-22c7-4f7c-9e73-8edcdb5a18e4" (UID: "d7343f75-22c7-4f7c-9e73-8edcdb5a18e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:05.333443 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.333345    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tmp-dir" pod "d7343f75-22c7-4f7c-9e73-8edcdb5a18e4" (UID: "d7343f75-22c7-4f7c-9e73-8edcdb5a18e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:05.333570 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.333551    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:05.333632 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.333574    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:05.333632 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.333586    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:05.335131 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.335110    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-dshm" pod "d7343f75-22c7-4f7c-9e73-8edcdb5a18e4" (UID: "d7343f75-22c7-4f7c-9e73-8edcdb5a18e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:05.335235 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.335174    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-kube-api-access-brllg" pod "d7343f75-22c7-4f7c-9e73-8edcdb5a18e4" (UID: "d7343f75-22c7-4f7c-9e73-8edcdb5a18e4"). InnerVolumeSpecName "kube-api-access-brllg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:05.335280 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.335230    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tls-certs" pod "d7343f75-22c7-4f7c-9e73-8edcdb5a18e4" (UID: "d7343f75-22c7-4f7c-9e73-8edcdb5a18e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:05.373317 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.373282    5663 generic.go:356] "Generic (PLEG): container finished" podID="d7343f75-22c7-4f7c-9e73-8edcdb5a18e4" containerID="88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45" exitCode=0
Oct 31 12:59:05.373481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.373326    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"
Oct 31 12:59:05.373481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.373379    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64" event={"ID":"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4","Type":"ContainerDied","Data":"88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45"}
Oct 31 12:59:05.373481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.373430    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64" event={"ID":"d7343f75-22c7-4f7c-9e73-8edcdb5a18e4","Type":"ContainerDied","Data":"7be7e111f054597e5872aa2e808acac0bcc07a50562cd84a13625e922b6cb9b6"}
Oct 31 12:59:05.373481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.373449    5663 scope.go:122] "RemoveContainer" containerID="88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45"
Oct 31 12:59:05.383180 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.383161    5663 scope.go:122] "RemoveContainer" containerID="88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45"
Oct 31 12:59:05.383475 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:05.383452    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45\": container with ID starting with 88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45 not found: ID does not exist" containerID="88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45"
Oct 31 12:59:05.383574 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.383482    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45"} err="failed to get container status \"88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45\": rpc error: code = NotFound desc = could not find container \"88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45\": container with ID starting with 88a6803b7624c57cfa19a302cc474e940aa788ee311db139b106ca97c215ba45 not found: ID does not exist"
Oct 31 12:59:05.396815 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.396792    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"]
Oct 31 12:59:05.401738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.401712    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fd988c/test-model-fd988c-kserve-6749b4d879-6wr64"]
Oct 31 12:59:05.434431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.434407    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brllg\" (UniqueName: \"kubernetes.io/projected/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-kube-api-access-brllg\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:05.434431 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.434431    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:05.434634 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.434446    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:05.438685 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:05.438662    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7343f75-22c7-4f7c-9e73-8edcdb5a18e4" path="/var/lib/kubelet/pods/d7343f75-22c7-4f7c-9e73-8edcdb5a18e4/volumes"
Oct 31 12:59:07.831195 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:07.831157    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7" podUID="71f3a982-4cac-4228-b891-dca75a989765" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:08.706521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:08.706482    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"]
Oct 31 12:59:08.706787 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:08.706740    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7" podUID="2ee276b8-0de2-4586-9a97-40cc65048ea5" containerName="maas-api" containerID="cri-o://3598f0f69399e39594b68420765568620debb5846aa6da0e04c5376e14943ffa" gracePeriod=30
Oct 31 12:59:09.013375 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.011405    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"]
Oct 31 12:59:09.031037 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.030986    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d176a5-db4df"]
Oct 31 12:59:09.440170 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.439606    5663 generic.go:356] "Generic (PLEG): container finished" podID="2ee276b8-0de2-4586-9a97-40cc65048ea5" containerID="3598f0f69399e39594b68420765568620debb5846aa6da0e04c5376e14943ffa" exitCode=0
Oct 31 12:59:09.447118 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.447079    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da38c69-768b-4569-b55e-43baf1f1e0bb" path="/var/lib/kubelet/pods/1da38c69-768b-4569-b55e-43baf1f1e0bb/volumes"
Oct 31 12:59:09.447505 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.447484    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7" event={"ID":"2ee276b8-0de2-4586-9a97-40cc65048ea5","Type":"ContainerDied","Data":"3598f0f69399e39594b68420765568620debb5846aa6da0e04c5376e14943ffa"}
Oct 31 12:59:09.447881 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.447855    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:59:09.524528 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.524491    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-maas-api-tls\") pod \"2ee276b8-0de2-4586-9a97-40cc65048ea5\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") "
Oct 31 12:59:09.524738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.524580    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ee276b8-0de2-4586-9a97-40cc65048ea5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee276b8-0de2-4586-9a97-40cc65048ea5-openshift-service-ca\") pod \"2ee276b8-0de2-4586-9a97-40cc65048ea5\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") "
Oct 31 12:59:09.524738 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.524666    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-metrics-certs\") pod \"2ee276b8-0de2-4586-9a97-40cc65048ea5\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") "
Oct 31 12:59:09.524876 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.524827    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ee276b8-0de2-4586-9a97-40cc65048ea5-kube-api-access-52c4w\" (UniqueName: \"kubernetes.io/projected/2ee276b8-0de2-4586-9a97-40cc65048ea5-kube-api-access-52c4w\") pod \"2ee276b8-0de2-4586-9a97-40cc65048ea5\" (UID: \"2ee276b8-0de2-4586-9a97-40cc65048ea5\") "
Oct 31 12:59:09.524990 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.524960    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee276b8-0de2-4586-9a97-40cc65048ea5-openshift-service-ca" pod "2ee276b8-0de2-4586-9a97-40cc65048ea5" (UID: "2ee276b8-0de2-4586-9a97-40cc65048ea5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:09.525488 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.525463    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ee276b8-0de2-4586-9a97-40cc65048ea5-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:09.527153 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.527129    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee276b8-0de2-4586-9a97-40cc65048ea5-kube-api-access-52c4w" pod "2ee276b8-0de2-4586-9a97-40cc65048ea5" (UID: "2ee276b8-0de2-4586-9a97-40cc65048ea5"). InnerVolumeSpecName "kube-api-access-52c4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:09.527252 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.527235    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-maas-api-tls" pod "2ee276b8-0de2-4586-9a97-40cc65048ea5" (UID: "2ee276b8-0de2-4586-9a97-40cc65048ea5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:09.527775 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.527749    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-metrics-certs" pod "2ee276b8-0de2-4586-9a97-40cc65048ea5" (UID: "2ee276b8-0de2-4586-9a97-40cc65048ea5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:09.627097 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.627019    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:09.627097 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.627049    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52c4w\" (UniqueName: \"kubernetes.io/projected/2ee276b8-0de2-4586-9a97-40cc65048ea5-kube-api-access-52c4w\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:09.627097 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:09.627060    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ee276b8-0de2-4586-9a97-40cc65048ea5-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:10.450946 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:10.448500    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"
Oct 31 12:59:10.450946 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:10.448517    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7" event={"ID":"2ee276b8-0de2-4586-9a97-40cc65048ea5","Type":"ContainerDied","Data":"a3b186cc003ce33e44668cb3b85d2fe7878b46697e1b6d7854395403c87dd41a"}
Oct 31 12:59:10.450946 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:10.448580    5663 scope.go:122] "RemoveContainer" containerID="3598f0f69399e39594b68420765568620debb5846aa6da0e04c5376e14943ffa"
Oct 31 12:59:10.452613 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:10.452582    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"]
Oct 31 12:59:10.452819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:10.452796    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" podUID="6e5f61f8-fc82-4778-a23e-5326e73fb615" containerName="istio-proxy" containerID="cri-o://d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83" gracePeriod=30
Oct 31 12:59:10.514031 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:10.513992    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"]
Oct 31 12:59:10.517449 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:10.517426    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d176a5-7b698689b-vknb7"]
Oct 31 12:59:11.277553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:11.276704    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"]
Oct 31 12:59:11.277553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:11.276973    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerName="istio-proxy" containerID="cri-o://59451011cbe7844a074c3d17f986c7b985ba344643b6d2a5225f95142dfb2c83" gracePeriod=30
Oct 31 12:59:11.280699 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:11.280674    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"]
Oct 31 12:59:11.280883 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:11.280860    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" podUID="16395ca3-67a4-48e6-8859-a941d5bec3f3" containerName="istio-proxy" containerID="cri-o://4b9fa27c813345e7214e9b95bd02d97de6bbc1108ff8e8ffdb95ef6415fd3859" gracePeriod=30
Oct 31 12:59:11.444009 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:11.443978    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ee276b8-0de2-4586-9a97-40cc65048ea5" path="/var/lib/kubelet/pods/2ee276b8-0de2-4586-9a97-40cc65048ea5/volumes"
Oct 31 12:59:12.554246 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:12.554211    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj" podUID="1ceb5e4b-0314-4f7e-bd59-ec883e6db112" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:15.664212 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.664150    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:59:15.665174 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.664689    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:59:15.680980 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.680945    5663 patch_prober.go:29] interesting pod/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:59:15.681163 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.680999    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" podUID="16395ca3-67a4-48e6-8859-a941d5bec3f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:59:15.720335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.720268    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:59:15.725802 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.725767    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-credential-socket\") pod \"6e5f61f8-fc82-4778-a23e-5326e73fb615\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") "
Oct 31 12:59:15.725926 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.725828    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-podinfo\") pod \"6e5f61f8-fc82-4778-a23e-5326e73fb615\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") "
Oct 31 12:59:15.725926 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.725868    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-envoy\") pod \"6e5f61f8-fc82-4778-a23e-5326e73fb615\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") "
Oct 31 12:59:15.725926 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.725886    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-certs\") pod \"6e5f61f8-fc82-4778-a23e-5326e73fb615\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") "
Oct 31 12:59:15.726026 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.725945    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-kube-api-access-d7wjx\" (UniqueName: \"kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-kube-api-access-d7wjx\") pod \"6e5f61f8-fc82-4778-a23e-5326e73fb615\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") "
Oct 31 12:59:15.726026 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726002    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-data\") pod \"6e5f61f8-fc82-4778-a23e-5326e73fb615\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") "
Oct 31 12:59:15.726112 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726036    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-credential-socket" pod "6e5f61f8-fc82-4778-a23e-5326e73fb615" (UID: "6e5f61f8-fc82-4778-a23e-5326e73fb615"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:15.726112 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726069    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-token\" (UniqueName: \"kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-token\") pod \"6e5f61f8-fc82-4778-a23e-5326e73fb615\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") "
Oct 31 12:59:15.726219 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726120    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-socket\") pod \"6e5f61f8-fc82-4778-a23e-5326e73fb615\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") "
Oct 31 12:59:15.726219 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726210    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e5f61f8-fc82-4778-a23e-5326e73fb615-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5f61f8-fc82-4778-a23e-5326e73fb615-istiod-ca-cert\") pod \"6e5f61f8-fc82-4778-a23e-5326e73fb615\" (UID: \"6e5f61f8-fc82-4778-a23e-5326e73fb615\") "
Oct 31 12:59:15.726325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726261    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-certs" pod "6e5f61f8-fc82-4778-a23e-5326e73fb615" (UID: "6e5f61f8-fc82-4778-a23e-5326e73fb615"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:15.726325 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726314    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-data" pod "6e5f61f8-fc82-4778-a23e-5326e73fb615" (UID: "6e5f61f8-fc82-4778-a23e-5326e73fb615"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:15.726514 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726489    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-socket" pod "6e5f61f8-fc82-4778-a23e-5326e73fb615" (UID: "6e5f61f8-fc82-4778-a23e-5326e73fb615"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:15.726870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726645    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5f61f8-fc82-4778-a23e-5326e73fb615-istiod-ca-cert" pod "6e5f61f8-fc82-4778-a23e-5326e73fb615" (UID: "6e5f61f8-fc82-4778-a23e-5326e73fb615"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:15.726870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726780    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.726870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726802    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.726870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726816    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.726870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726831    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5f61f8-fc82-4778-a23e-5326e73fb615-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.726870 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.726846    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.728261 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.728229    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-kube-api-access-d7wjx" pod "6e5f61f8-fc82-4778-a23e-5326e73fb615" (UID: "6e5f61f8-fc82-4778-a23e-5326e73fb615"). InnerVolumeSpecName "kube-api-access-d7wjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:15.728384 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.728266    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-podinfo" pod "6e5f61f8-fc82-4778-a23e-5326e73fb615" (UID: "6e5f61f8-fc82-4778-a23e-5326e73fb615"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:15.728524 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.728506    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-token" pod "6e5f61f8-fc82-4778-a23e-5326e73fb615" (UID: "6e5f61f8-fc82-4778-a23e-5326e73fb615"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:15.728585 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.728553    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-envoy" pod "6e5f61f8-fc82-4778-a23e-5326e73fb615" (UID: "6e5f61f8-fc82-4778-a23e-5326e73fb615"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:15.828592 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.828558    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7wjx\" (UniqueName: \"kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-kube-api-access-d7wjx\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.828592 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.828588    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.828592 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.828599    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:15.828837 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:15.828608    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5f61f8-fc82-4778-a23e-5326e73fb615-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.493133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.493093    5663 generic.go:356] "Generic (PLEG): container finished" podID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerID="59451011cbe7844a074c3d17f986c7b985ba344643b6d2a5225f95142dfb2c83" exitCode=0
Oct 31 12:59:16.493299 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.493240    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" event={"ID":"ae245313-05e0-4c03-aa7f-3d2e9eabf675","Type":"ContainerDied","Data":"59451011cbe7844a074c3d17f986c7b985ba344643b6d2a5225f95142dfb2c83"}
Oct 31 12:59:16.495717 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.495687    5663 generic.go:356] "Generic (PLEG): container finished" podID="16395ca3-67a4-48e6-8859-a941d5bec3f3" containerID="4b9fa27c813345e7214e9b95bd02d97de6bbc1108ff8e8ffdb95ef6415fd3859" exitCode=0
Oct 31 12:59:16.495871 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.495769    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" event={"ID":"16395ca3-67a4-48e6-8859-a941d5bec3f3","Type":"ContainerDied","Data":"4b9fa27c813345e7214e9b95bd02d97de6bbc1108ff8e8ffdb95ef6415fd3859"}
Oct 31 12:59:16.497613 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.497586    5663 generic.go:356] "Generic (PLEG): container finished" podID="6e5f61f8-fc82-4778-a23e-5326e73fb615" containerID="d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83" exitCode=0
Oct 31 12:59:16.497734 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.497670    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" event={"ID":"6e5f61f8-fc82-4778-a23e-5326e73fb615","Type":"ContainerDied","Data":"d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83"}
Oct 31 12:59:16.497734 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.497711    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp" event={"ID":"6e5f61f8-fc82-4778-a23e-5326e73fb615","Type":"ContainerDied","Data":"64cfd10743b1dc72b77aedf7dcab05b693c16f2520a5c8ecac8ad488a7ca5f95"}
Oct 31 12:59:16.497838 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.497744    5663 scope.go:122] "RemoveContainer" containerID="d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83"
Oct 31 12:59:16.497913 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.497897    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"
Oct 31 12:59:16.513408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.513364    5663 scope.go:122] "RemoveContainer" containerID="d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83"
Oct 31 12:59:16.513770 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:16.513736    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83\": container with ID starting with d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83 not found: ID does not exist" containerID="d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83"
Oct 31 12:59:16.513852 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.513781    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83"} err="failed to get container status \"d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83\": rpc error: code = NotFound desc = could not find container \"d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83\": container with ID starting with d74f31e3dc62fbe8214fd6569d421ab2546de35160f40ad3202845d8e71d3a83 not found: ID does not exist"
Oct 31 12:59:16.558278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.556486    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"]
Oct 31 12:59:16.568861 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.568822    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-7n6sp"]
Oct 31 12:59:16.649900 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.649870    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:59:16.651798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.651774    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:59:16.760437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760406    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-certs\") pod \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") "
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760445    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istiod-ca-cert\") pod \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") "
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760465    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-kube-api-access-qbrxc\" (UniqueName: \"kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-kube-api-access-qbrxc\") pod \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") "
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760499    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-podinfo\") pod \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") "
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760605    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16395ca3-67a4-48e6-8859-a941d5bec3f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16395ca3-67a4-48e6-8859-a941d5bec3f3-istiod-ca-cert\") pod \"16395ca3-67a4-48e6-8859-a941d5bec3f3\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") "
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760635    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-envoy\") pod \"16395ca3-67a4-48e6-8859-a941d5bec3f3\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") "
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760664    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-token\" (UniqueName: \"kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-token\") pod \"16395ca3-67a4-48e6-8859-a941d5bec3f3\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") "
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760697    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-podinfo\") pod \"16395ca3-67a4-48e6-8859-a941d5bec3f3\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") "
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760742    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-data\") pod \"16395ca3-67a4-48e6-8859-a941d5bec3f3\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") "
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760758    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-certs" pod "ae245313-05e0-4c03-aa7f-3d2e9eabf675" (UID: "ae245313-05e0-4c03-aa7f-3d2e9eabf675"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760785    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-credential-socket\") pod \"16395ca3-67a4-48e6-8859-a941d5bec3f3\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") "
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760867    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-socket\") pod \"16395ca3-67a4-48e6-8859-a941d5bec3f3\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") "
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760925    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istiod-ca-cert" pod "ae245313-05e0-4c03-aa7f-3d2e9eabf675" (UID: "ae245313-05e0-4c03-aa7f-3d2e9eabf675"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:16.760934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.760934    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-credential-socket\") pod \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") "
Oct 31 12:59:16.761737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761004    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-token\" (UniqueName: \"kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-token\") pod \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") "
Oct 31 12:59:16.761737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761046    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-kube-api-access-ph26k\" (UniqueName: \"kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-kube-api-access-ph26k\") pod \"16395ca3-67a4-48e6-8859-a941d5bec3f3\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") "
Oct 31 12:59:16.761737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761068    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-credential-socket" pod "16395ca3-67a4-48e6-8859-a941d5bec3f3" (UID: "16395ca3-67a4-48e6-8859-a941d5bec3f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.761737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761091    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-envoy\") pod \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") "
Oct 31 12:59:16.761737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761103    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16395ca3-67a4-48e6-8859-a941d5bec3f3-istiod-ca-cert" pod "16395ca3-67a4-48e6-8859-a941d5bec3f3" (UID: "16395ca3-67a4-48e6-8859-a941d5bec3f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:16.761737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761126    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-data\") pod \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") "
Oct 31 12:59:16.761737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761198    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-certs\") pod \"16395ca3-67a4-48e6-8859-a941d5bec3f3\" (UID: \"16395ca3-67a4-48e6-8859-a941d5bec3f3\") "
Oct 31 12:59:16.761737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761238    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-socket\") pod \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\" (UID: \"ae245313-05e0-4c03-aa7f-3d2e9eabf675\") "
Oct 31 12:59:16.761737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761257    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-data" pod "16395ca3-67a4-48e6-8859-a941d5bec3f3" (UID: "16395ca3-67a4-48e6-8859-a941d5bec3f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.761737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761368    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-socket" pod "16395ca3-67a4-48e6-8859-a941d5bec3f3" (UID: "16395ca3-67a4-48e6-8859-a941d5bec3f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.761737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761515    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-credential-socket" pod "ae245313-05e0-4c03-aa7f-3d2e9eabf675" (UID: "ae245313-05e0-4c03-aa7f-3d2e9eabf675"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.761737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761710    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-socket" pod "ae245313-05e0-4c03-aa7f-3d2e9eabf675" (UID: "ae245313-05e0-4c03-aa7f-3d2e9eabf675"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.762408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761888    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-data" pod "ae245313-05e0-4c03-aa7f-3d2e9eabf675" (UID: "ae245313-05e0-4c03-aa7f-3d2e9eabf675"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.762408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.761929    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-certs" pod "16395ca3-67a4-48e6-8859-a941d5bec3f3" (UID: "16395ca3-67a4-48e6-8859-a941d5bec3f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.762408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.762248    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16395ca3-67a4-48e6-8859-a941d5bec3f3-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.762408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.762271    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.762408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.762289    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.762408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.762306    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.762408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.762321    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.762408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.762336    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.762408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.762367    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.762408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.762383    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.762408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.762397    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.762408 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.762411    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.763727 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.763696    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-token" pod "ae245313-05e0-4c03-aa7f-3d2e9eabf675" (UID: "ae245313-05e0-4c03-aa7f-3d2e9eabf675"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:16.763812 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.763730    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-envoy" pod "16395ca3-67a4-48e6-8859-a941d5bec3f3" (UID: "16395ca3-67a4-48e6-8859-a941d5bec3f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.764342 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.764316    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-token" pod "16395ca3-67a4-48e6-8859-a941d5bec3f3" (UID: "16395ca3-67a4-48e6-8859-a941d5bec3f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:16.764470 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.764336    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-envoy" pod "ae245313-05e0-4c03-aa7f-3d2e9eabf675" (UID: "ae245313-05e0-4c03-aa7f-3d2e9eabf675"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:16.764470 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.764383    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-kube-api-access-qbrxc" pod "ae245313-05e0-4c03-aa7f-3d2e9eabf675" (UID: "ae245313-05e0-4c03-aa7f-3d2e9eabf675"). InnerVolumeSpecName "kube-api-access-qbrxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:16.764470 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.764323    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-kube-api-access-ph26k" pod "16395ca3-67a4-48e6-8859-a941d5bec3f3" (UID: "16395ca3-67a4-48e6-8859-a941d5bec3f3"). InnerVolumeSpecName "kube-api-access-ph26k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:16.764671 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.764510    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-podinfo" pod "ae245313-05e0-4c03-aa7f-3d2e9eabf675" (UID: "ae245313-05e0-4c03-aa7f-3d2e9eabf675"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:16.764990 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.764974    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-podinfo" pod "16395ca3-67a4-48e6-8859-a941d5bec3f3" (UID: "16395ca3-67a4-48e6-8859-a941d5bec3f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:16.863313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.863230    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.863313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.863259    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.863313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.863268    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16395ca3-67a4-48e6-8859-a941d5bec3f3-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.863313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.863279    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.863313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.863288    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph26k\" (UniqueName: \"kubernetes.io/projected/16395ca3-67a4-48e6-8859-a941d5bec3f3-kube-api-access-ph26k\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.863313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.863298    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.863313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.863307    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbrxc\" (UniqueName: \"kubernetes.io/projected/ae245313-05e0-4c03-aa7f-3d2e9eabf675-kube-api-access-qbrxc\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:16.863313 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:16.863316    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae245313-05e0-4c03-aa7f-3d2e9eabf675-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:17.438922 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.438884    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e5f61f8-fc82-4778-a23e-5326e73fb615" path="/var/lib/kubelet/pods/6e5f61f8-fc82-4778-a23e-5326e73fb615/volumes"
Oct 31 12:59:17.520243 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.520203    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6" event={"ID":"ae245313-05e0-4c03-aa7f-3d2e9eabf675","Type":"ContainerDied","Data":"24417806c25ab1343f3923b31f1394ffe4da91c5e3f3630e8ac454382c8154d5"}
Oct 31 12:59:17.520522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.520281    5663 scope.go:122] "RemoveContainer" containerID="59451011cbe7844a074c3d17f986c7b985ba344643b6d2a5225f95142dfb2c83"
Oct 31 12:59:17.520522 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.520473    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"
Oct 31 12:59:17.525554 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.524189    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t" event={"ID":"16395ca3-67a4-48e6-8859-a941d5bec3f3","Type":"ContainerDied","Data":"ffce4ac63845bc72480ec9cac505625c902b4b333fb64830dfe842f9af11a0ba"}
Oct 31 12:59:17.525554 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.524347    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"
Oct 31 12:59:17.605662 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.598799    5663 scope.go:122] "RemoveContainer" containerID="4b9fa27c813345e7214e9b95bd02d97de6bbc1108ff8e8ffdb95ef6415fd3859"
Oct 31 12:59:17.612379 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.611119    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"]
Oct 31 12:59:17.615797 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.615763    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-4p48t"]
Oct 31 12:59:17.638531 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.638495    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"]
Oct 31 12:59:17.640624 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.640599    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d086f3-openshift-default-5c7884ccc-sd4k6"]
Oct 31 12:59:17.829200 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.829170    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"]
Oct 31 12:59:17.830609 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.830584    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7" podUID="71f3a982-4cac-4228-b891-dca75a989765" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:17.863484 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:17.863447    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"]
Oct 31 12:59:19.221094 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.221070    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"
Oct 31 12:59:19.317787 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.317706    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f3a982-4cac-4228-b891-dca75a989765-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f3a982-4cac-4228-b891-dca75a989765-plugins-config-volume\") pod \"71f3a982-4cac-4228-b891-dca75a989765\" (UID: \"71f3a982-4cac-4228-b891-dca75a989765\") "
Oct 31 12:59:19.317958 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.317802    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f3a982-4cac-4228-b891-dca75a989765-kube-api-access-z5phd\" (UniqueName: \"kubernetes.io/projected/71f3a982-4cac-4228-b891-dca75a989765-kube-api-access-z5phd\") pod \"71f3a982-4cac-4228-b891-dca75a989765\" (UID: \"71f3a982-4cac-4228-b891-dca75a989765\") "
Oct 31 12:59:19.318139 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.318115    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f3a982-4cac-4228-b891-dca75a989765-plugins-config-volume" pod "71f3a982-4cac-4228-b891-dca75a989765" (UID: "71f3a982-4cac-4228-b891-dca75a989765"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:19.318322 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.318308    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f3a982-4cac-4228-b891-dca75a989765-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:19.319811 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.319787    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f3a982-4cac-4228-b891-dca75a989765-kube-api-access-z5phd" pod "71f3a982-4cac-4228-b891-dca75a989765" (UID: "71f3a982-4cac-4228-b891-dca75a989765"). InnerVolumeSpecName "kube-api-access-z5phd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:19.419291 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.419256    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5phd\" (UniqueName: \"kubernetes.io/projected/71f3a982-4cac-4228-b891-dca75a989765-kube-api-access-z5phd\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:19.439363 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.439331    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16395ca3-67a4-48e6-8859-a941d5bec3f3" path="/var/lib/kubelet/pods/16395ca3-67a4-48e6-8859-a941d5bec3f3/volumes"
Oct 31 12:59:19.439865 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.439848    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" path="/var/lib/kubelet/pods/ae245313-05e0-4c03-aa7f-3d2e9eabf675/volumes"
Oct 31 12:59:19.539333 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.539301    5663 generic.go:356] "Generic (PLEG): container finished" podID="71f3a982-4cac-4228-b891-dca75a989765" containerID="9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3" exitCode=0
Oct 31 12:59:19.539539 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.539361    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"
Oct 31 12:59:19.539539 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.539407    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7" event={"ID":"71f3a982-4cac-4228-b891-dca75a989765","Type":"ContainerDied","Data":"9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3"}
Oct 31 12:59:19.539539 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.539460    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7" event={"ID":"71f3a982-4cac-4228-b891-dca75a989765","Type":"ContainerDied","Data":"9999e7dedb9938708ba7c34925d52101b22a0f7b231023a8133ccafb6781d670"}
Oct 31 12:59:19.539539 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.539483    5663 scope.go:122] "RemoveContainer" containerID="9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3"
Oct 31 12:59:19.549148 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.549131    5663 scope.go:122] "RemoveContainer" containerID="9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3"
Oct 31 12:59:19.549498 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:19.549478    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3\": container with ID starting with 9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3 not found: ID does not exist" containerID="9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3"
Oct 31 12:59:19.549586 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.549503    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3"} err="failed to get container status \"9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3\": rpc error: code = NotFound desc = could not find container \"9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3\": container with ID starting with 9a60a06fa032f51025ffdb1b1579fde06d9696f48dac02e77421727dd37456e3 not found: ID does not exist"
Oct 31 12:59:19.564502 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.564479    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"]
Oct 31 12:59:19.566789 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:19.566767    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d086f3-858b7cccd7-mvnq7"]
Oct 31 12:59:21.439877 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:21.439838    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f3a982-4cac-4228-b891-dca75a989765" path="/var/lib/kubelet/pods/71f3a982-4cac-4228-b891-dca75a989765/volumes"
Oct 31 12:59:21.878194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:21.878156    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"]
Oct 31 12:59:22.476721 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:22.476683    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:59:22.549206 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:22.549170    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj" podUID="1ceb5e4b-0314-4f7e-bd59-ec883e6db112" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:22.829597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:22.829552    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts" podUID="9c58e1e6-909c-47d7-98f1-508b00de8fd5" containerName="payload-processing" containerID="cri-o://ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac" gracePeriod=25
Oct 31 12:59:22.864382 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:22.864330    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt" podUID="aa472831-ecc0-4dbb-9bcf-4aec35346832" containerName="payload-pre-processing" containerID="cri-o://13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180" gracePeriod=25
Oct 31 12:59:23.105693 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.105666    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"
Oct 31 12:59:23.190936 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.190907    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c58e1e6-909c-47d7-98f1-508b00de8fd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c58e1e6-909c-47d7-98f1-508b00de8fd5-plugins-config-volume\") pod \"9c58e1e6-909c-47d7-98f1-508b00de8fd5\" (UID: \"9c58e1e6-909c-47d7-98f1-508b00de8fd5\") "
Oct 31 12:59:23.191102 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.191020    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c58e1e6-909c-47d7-98f1-508b00de8fd5-kube-api-access-q4hzz\" (UniqueName: \"kubernetes.io/projected/9c58e1e6-909c-47d7-98f1-508b00de8fd5-kube-api-access-q4hzz\") pod \"9c58e1e6-909c-47d7-98f1-508b00de8fd5\" (UID: \"9c58e1e6-909c-47d7-98f1-508b00de8fd5\") "
Oct 31 12:59:23.191274 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.191252    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c58e1e6-909c-47d7-98f1-508b00de8fd5-plugins-config-volume" pod "9c58e1e6-909c-47d7-98f1-508b00de8fd5" (UID: "9c58e1e6-909c-47d7-98f1-508b00de8fd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:23.191481 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.191468    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c58e1e6-909c-47d7-98f1-508b00de8fd5-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:23.193128 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.193106    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c58e1e6-909c-47d7-98f1-508b00de8fd5-kube-api-access-q4hzz" pod "9c58e1e6-909c-47d7-98f1-508b00de8fd5" (UID: "9c58e1e6-909c-47d7-98f1-508b00de8fd5"). InnerVolumeSpecName "kube-api-access-q4hzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:23.223692 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.223650    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"]
Oct 31 12:59:23.223934 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.223905    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb" podUID="1f09a69d-29df-4586-8836-49bdc029a99b" containerName="maas-api" containerID="cri-o://6615b6491bf939288f4b3d178572e009483523234262be9f6bc92cc8cf3e452e" gracePeriod=30
Oct 31 12:59:23.292703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.292666    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4hzz\" (UniqueName: \"kubernetes.io/projected/9c58e1e6-909c-47d7-98f1-508b00de8fd5-kube-api-access-q4hzz\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:23.601147 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.601062    5663 generic.go:356] "Generic (PLEG): container finished" podID="1f09a69d-29df-4586-8836-49bdc029a99b" containerID="6615b6491bf939288f4b3d178572e009483523234262be9f6bc92cc8cf3e452e" exitCode=0
Oct 31 12:59:23.601909 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.601835    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb" event={"ID":"1f09a69d-29df-4586-8836-49bdc029a99b","Type":"ContainerDied","Data":"6615b6491bf939288f4b3d178572e009483523234262be9f6bc92cc8cf3e452e"}
Oct 31 12:59:23.604075 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.604046    5663 generic.go:356] "Generic (PLEG): container finished" podID="9c58e1e6-909c-47d7-98f1-508b00de8fd5" containerID="ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac" exitCode=0
Oct 31 12:59:23.604418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.604344    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts" event={"ID":"9c58e1e6-909c-47d7-98f1-508b00de8fd5","Type":"ContainerDied","Data":"ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac"}
Oct 31 12:59:23.604608 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.604584    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts" event={"ID":"9c58e1e6-909c-47d7-98f1-508b00de8fd5","Type":"ContainerDied","Data":"e43f3e10513e04c2e0a3016842c55f5486a574e99ae346e810987ce3c43036f4"}
Oct 31 12:59:23.604743 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.604731    5663 scope.go:122] "RemoveContainer" containerID="ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac"
Oct 31 12:59:23.604991 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.604972    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"
Oct 31 12:59:23.652208 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.651406    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"]
Oct 31 12:59:23.653122 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.652405    5663 scope.go:122] "RemoveContainer" containerID="ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac"
Oct 31 12:59:23.653122 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:23.653036    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac\": container with ID starting with ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac not found: ID does not exist" containerID="ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac"
Oct 31 12:59:23.653122 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.653074    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac"} err="failed to get container status \"ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac\": rpc error: code = NotFound desc = could not find container \"ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac\": container with ID starting with ac33a92a8d8b1900803b73e53dba39cffb5db84e9722bf799abb593e5b2adaac not found: ID does not exist"
Oct 31 12:59:23.654305 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.654284    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2c393f-56cc894cdc-4h6ts"]
Oct 31 12:59:23.663810 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.663793    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:59:23.802377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.802224    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-metrics-certs\") pod \"1f09a69d-29df-4586-8836-49bdc029a99b\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") "
Oct 31 12:59:23.802377 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.802340    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-maas-api-tls\") pod \"1f09a69d-29df-4586-8836-49bdc029a99b\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") "
Oct 31 12:59:23.802737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.802528    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f09a69d-29df-4586-8836-49bdc029a99b-kube-api-access-vsqvb\" (UniqueName: \"kubernetes.io/projected/1f09a69d-29df-4586-8836-49bdc029a99b-kube-api-access-vsqvb\") pod \"1f09a69d-29df-4586-8836-49bdc029a99b\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") "
Oct 31 12:59:23.802737 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.802658    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f09a69d-29df-4586-8836-49bdc029a99b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f09a69d-29df-4586-8836-49bdc029a99b-openshift-service-ca\") pod \"1f09a69d-29df-4586-8836-49bdc029a99b\" (UID: \"1f09a69d-29df-4586-8836-49bdc029a99b\") "
Oct 31 12:59:23.803393 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.803343    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f09a69d-29df-4586-8836-49bdc029a99b-openshift-service-ca" pod "1f09a69d-29df-4586-8836-49bdc029a99b" (UID: "1f09a69d-29df-4586-8836-49bdc029a99b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:23.804823 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.804792    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f09a69d-29df-4586-8836-49bdc029a99b-kube-api-access-vsqvb" pod "1f09a69d-29df-4586-8836-49bdc029a99b" (UID: "1f09a69d-29df-4586-8836-49bdc029a99b"). InnerVolumeSpecName "kube-api-access-vsqvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:23.804939 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.804852    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-maas-api-tls" pod "1f09a69d-29df-4586-8836-49bdc029a99b" (UID: "1f09a69d-29df-4586-8836-49bdc029a99b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:23.804939 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.804889    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-metrics-certs" pod "1f09a69d-29df-4586-8836-49bdc029a99b" (UID: "1f09a69d-29df-4586-8836-49bdc029a99b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:23.904834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.904758    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f09a69d-29df-4586-8836-49bdc029a99b-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:23.904834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.904787    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:23.904834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.904796    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f09a69d-29df-4586-8836-49bdc029a99b-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:23.904834 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:23.904805    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsqvb\" (UniqueName: \"kubernetes.io/projected/1f09a69d-29df-4586-8836-49bdc029a99b-kube-api-access-vsqvb\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:24.613241 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:24.613201    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb" event={"ID":"1f09a69d-29df-4586-8836-49bdc029a99b","Type":"ContainerDied","Data":"ca672e7cf112d8770ef230254886c5cb7cc9aea92076ae3bcc40a159466d5b45"}
Oct 31 12:59:24.613241 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:24.613233    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"
Oct 31 12:59:24.613798 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:24.613266    5663 scope.go:122] "RemoveContainer" containerID="6615b6491bf939288f4b3d178572e009483523234262be9f6bc92cc8cf3e452e"
Oct 31 12:59:24.636942 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:24.636912    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"]
Oct 31 12:59:24.642913 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:24.642890    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-71bc05-f6bc95799-dx2xb"]
Oct 31 12:59:25.440964 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.440908    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f09a69d-29df-4586-8836-49bdc029a99b" path="/var/lib/kubelet/pods/1f09a69d-29df-4586-8836-49bdc029a99b/volumes"
Oct 31 12:59:25.441595 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.441562    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c58e1e6-909c-47d7-98f1-508b00de8fd5" path="/var/lib/kubelet/pods/9c58e1e6-909c-47d7-98f1-508b00de8fd5/volumes"
Oct 31 12:59:25.510288 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.510232    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"
Oct 31 12:59:25.622248 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.622211    5663 generic.go:356] "Generic (PLEG): container finished" podID="1ceb5e4b-0314-4f7e-bd59-ec883e6db112" containerID="2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92" exitCode=0
Oct 31 12:59:25.622696 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.622272    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"
Oct 31 12:59:25.622696 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.622300    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj" event={"ID":"1ceb5e4b-0314-4f7e-bd59-ec883e6db112","Type":"ContainerDied","Data":"2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92"}
Oct 31 12:59:25.622696 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.622408    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj" event={"ID":"1ceb5e4b-0314-4f7e-bd59-ec883e6db112","Type":"ContainerDied","Data":"789583f193ff66eff5ad33ee574d73828f888123ff51af3d488dcbbbabf9fc10"}
Oct 31 12:59:25.622696 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.622447    5663 scope.go:122] "RemoveContainer" containerID="2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92"
Oct 31 12:59:25.626753 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.626728    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-plugins-config-volume\") pod \"1ceb5e4b-0314-4f7e-bd59-ec883e6db112\" (UID: \"1ceb5e4b-0314-4f7e-bd59-ec883e6db112\") "
Oct 31 12:59:25.626904 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.626867    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-kube-api-access-mf9v6\" (UniqueName: \"kubernetes.io/projected/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-kube-api-access-mf9v6\") pod \"1ceb5e4b-0314-4f7e-bd59-ec883e6db112\" (UID: \"1ceb5e4b-0314-4f7e-bd59-ec883e6db112\") "
Oct 31 12:59:25.627157 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.627131    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-plugins-config-volume" pod "1ceb5e4b-0314-4f7e-bd59-ec883e6db112" (UID: "1ceb5e4b-0314-4f7e-bd59-ec883e6db112"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:25.627262 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.627248    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:25.629177 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.629156    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-kube-api-access-mf9v6" pod "1ceb5e4b-0314-4f7e-bd59-ec883e6db112" (UID: "1ceb5e4b-0314-4f7e-bd59-ec883e6db112"). InnerVolumeSpecName "kube-api-access-mf9v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:25.634750 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.634726    5663 scope.go:122] "RemoveContainer" containerID="2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92"
Oct 31 12:59:25.635077 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:25.635047    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92\": container with ID starting with 2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92 not found: ID does not exist" containerID="2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92"
Oct 31 12:59:25.635160 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.635079    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92"} err="failed to get container status \"2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92\": rpc error: code = NotFound desc = could not find container \"2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92\": container with ID starting with 2c918f4c1f1fa80f248ec32860ea7ef93adbc74ba4cb6ff93e775a604899bd92 not found: ID does not exist"
Oct 31 12:59:25.728941 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.728909    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf9v6\" (UniqueName: \"kubernetes.io/projected/1ceb5e4b-0314-4f7e-bd59-ec883e6db112-kube-api-access-mf9v6\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:25.946469 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.946438    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"]
Oct 31 12:59:25.954385 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:25.954346    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d176a5-86b4dcc677-wcqwj"]
Oct 31 12:59:26.879017 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:26.878954    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx" podUID="a3328d1a-9f2b-42de-a5b6-35ad852a0d07" containerName="payload-processing" containerID="cri-o://38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072" gracePeriod=25
Oct 31 12:59:27.151585 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.151561    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"
Oct 31 12:59:27.246318 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.246273    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-plugins-config-volume\") pod \"a3328d1a-9f2b-42de-a5b6-35ad852a0d07\" (UID: \"a3328d1a-9f2b-42de-a5b6-35ad852a0d07\") "
Oct 31 12:59:27.246573 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.246339    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-kube-api-access-gcjql\" (UniqueName: \"kubernetes.io/projected/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-kube-api-access-gcjql\") pod \"a3328d1a-9f2b-42de-a5b6-35ad852a0d07\" (UID: \"a3328d1a-9f2b-42de-a5b6-35ad852a0d07\") "
Oct 31 12:59:27.247767 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.247734    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-plugins-config-volume" pod "a3328d1a-9f2b-42de-a5b6-35ad852a0d07" (UID: "a3328d1a-9f2b-42de-a5b6-35ad852a0d07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:27.278862 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.278827    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-kube-api-access-gcjql" pod "a3328d1a-9f2b-42de-a5b6-35ad852a0d07" (UID: "a3328d1a-9f2b-42de-a5b6-35ad852a0d07"). InnerVolumeSpecName "kube-api-access-gcjql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:27.347729 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.347695    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:27.347729 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.347722    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcjql\" (UniqueName: \"kubernetes.io/projected/a3328d1a-9f2b-42de-a5b6-35ad852a0d07-kube-api-access-gcjql\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:27.438649 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.438578    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ceb5e4b-0314-4f7e-bd59-ec883e6db112" path="/var/lib/kubelet/pods/1ceb5e4b-0314-4f7e-bd59-ec883e6db112/volumes"
Oct 31 12:59:27.636766 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.636731    5663 generic.go:356] "Generic (PLEG): container finished" podID="a3328d1a-9f2b-42de-a5b6-35ad852a0d07" containerID="38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072" exitCode=0
Oct 31 12:59:27.636973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.636785    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"
Oct 31 12:59:27.636973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.636815    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx" event={"ID":"a3328d1a-9f2b-42de-a5b6-35ad852a0d07","Type":"ContainerDied","Data":"38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072"}
Oct 31 12:59:27.636973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.636868    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx" event={"ID":"a3328d1a-9f2b-42de-a5b6-35ad852a0d07","Type":"ContainerDied","Data":"23e36e1b906d7c9da087e007b2387874a62af1922c5a8600c5604856b28417a8"}
Oct 31 12:59:27.636973 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.636891    5663 scope.go:122] "RemoveContainer" containerID="38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072"
Oct 31 12:59:27.646376 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.646345    5663 scope.go:122] "RemoveContainer" containerID="38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072"
Oct 31 12:59:27.646692 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:27.646668    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072\": container with ID starting with 38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072 not found: ID does not exist" containerID="38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072"
Oct 31 12:59:27.646782 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.646700    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072"} err="failed to get container status \"38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072\": rpc error: code = NotFound desc = could not find container \"38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072\": container with ID starting with 38140cdca01bf0629087751b6b68ce11b36b175f8055aec41857e8b911d48072 not found: ID does not exist"
Oct 31 12:59:27.658546 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.658518    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"]
Oct 31 12:59:27.660692 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:27.660670    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-71bc05-68b66778cf-2r7sx"]
Oct 31 12:59:28.116023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:28.115994    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"]
Oct 31 12:59:28.132418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:28.132341    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"]
Oct 31 12:59:29.438812 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:29.438780    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3328d1a-9f2b-42de-a5b6-35ad852a0d07" path="/var/lib/kubelet/pods/a3328d1a-9f2b-42de-a5b6-35ad852a0d07/volumes"
Oct 31 12:59:30.445644 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:30.445605    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt" podUID="aa472831-ecc0-4dbb-9bcf-4aec35346832" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:32.109277 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109241    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"]
Oct 31 12:59:32.109835 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109813    5663 cpu_manager.go:414] "containerMap: removing container" podUID="1da38c69-768b-4569-b55e-43baf1f1e0bb" containerName="revoke-keys"
Oct 31 12:59:32.109835 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109835    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da38c69-768b-4569-b55e-43baf1f1e0bb" containerName="revoke-keys"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109846    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d5b51a9d-7551-43d7-85a1-8a61abd5cfc9" containerName="main"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109851    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b51a9d-7551-43d7-85a1-8a61abd5cfc9" containerName="main"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109867    5663 cpu_manager.go:414] "containerMap: removing container" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerName="istio-proxy"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109873    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerName="istio-proxy"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109885    5663 cpu_manager.go:414] "containerMap: removing container" podUID="3c7e399c-2f1e-4d61-8116-68b897e31ed1" containerName="main"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109889    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7e399c-2f1e-4d61-8116-68b897e31ed1" containerName="main"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109897    5663 cpu_manager.go:414] "containerMap: removing container" podUID="6e5f61f8-fc82-4778-a23e-5326e73fb615" containerName="istio-proxy"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109903    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5f61f8-fc82-4778-a23e-5326e73fb615" containerName="istio-proxy"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109913    5663 cpu_manager.go:414] "containerMap: removing container" podUID="1ceb5e4b-0314-4f7e-bd59-ec883e6db112" containerName="payload-processing"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109918    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ceb5e4b-0314-4f7e-bd59-ec883e6db112" containerName="payload-processing"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109924    5663 cpu_manager.go:414] "containerMap: removing container" podUID="16395ca3-67a4-48e6-8859-a941d5bec3f3" containerName="istio-proxy"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109928    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16395ca3-67a4-48e6-8859-a941d5bec3f3" containerName="istio-proxy"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109935    5663 cpu_manager.go:414] "containerMap: removing container" podUID="2ee276b8-0de2-4586-9a97-40cc65048ea5" containerName="maas-api"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109939    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ee276b8-0de2-4586-9a97-40cc65048ea5" containerName="maas-api"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109948    5663 cpu_manager.go:414] "containerMap: removing container" podUID="71f3a982-4cac-4228-b891-dca75a989765" containerName="payload-pre-processing"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109952    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f3a982-4cac-4228-b891-dca75a989765" containerName="payload-pre-processing"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109958    5663 cpu_manager.go:414] "containerMap: removing container" podUID="9c58e1e6-909c-47d7-98f1-508b00de8fd5" containerName="payload-processing"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109964    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c58e1e6-909c-47d7-98f1-508b00de8fd5" containerName="payload-processing"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109971    5663 cpu_manager.go:414] "containerMap: removing container" podUID="a3328d1a-9f2b-42de-a5b6-35ad852a0d07" containerName="payload-processing"
Oct 31 12:59:32.109999 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.109978    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3328d1a-9f2b-42de-a5b6-35ad852a0d07" containerName="payload-processing"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110015    5663 cpu_manager.go:414] "containerMap: removing container" podUID="1f09a69d-29df-4586-8836-49bdc029a99b" containerName="maas-api"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110020    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f09a69d-29df-4586-8836-49bdc029a99b" containerName="maas-api"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110030    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d7343f75-22c7-4f7c-9e73-8edcdb5a18e4" containerName="main"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110035    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7343f75-22c7-4f7c-9e73-8edcdb5a18e4" containerName="main"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110115    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7343f75-22c7-4f7c-9e73-8edcdb5a18e4" containerName="main"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110132    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da38c69-768b-4569-b55e-43baf1f1e0bb" containerName="revoke-keys"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110139    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b51a9d-7551-43d7-85a1-8a61abd5cfc9" containerName="main"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110150    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae245313-05e0-4c03-aa7f-3d2e9eabf675" containerName="istio-proxy"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110162    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7e399c-2f1e-4d61-8116-68b897e31ed1" containerName="main"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110171    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5f61f8-fc82-4778-a23e-5326e73fb615" containerName="istio-proxy"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110181    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ceb5e4b-0314-4f7e-bd59-ec883e6db112" containerName="payload-processing"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110189    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="16395ca3-67a4-48e6-8859-a941d5bec3f3" containerName="istio-proxy"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110195    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ee276b8-0de2-4586-9a97-40cc65048ea5" containerName="maas-api"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110202    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f3a982-4cac-4228-b891-dca75a989765" containerName="payload-pre-processing"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110207    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c58e1e6-909c-47d7-98f1-508b00de8fd5" containerName="payload-processing"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110213    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3328d1a-9f2b-42de-a5b6-35ad852a0d07" containerName="payload-processing"
Oct 31 12:59:32.110703 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.110219    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f09a69d-29df-4586-8836-49bdc029a99b" containerName="maas-api"
Oct 31 12:59:32.114783 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.114766    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:32.128515 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.128492    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"]
Oct 31 12:59:32.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.172490    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:59:32.172689 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.172673    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:59:32.223074 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.223037    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9\" (UID: \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:32.223074 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.223071    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcnkc\" (UniqueName: \"kubernetes.io/projected/e9f21b3e-2cae-46db-9e45-58530e9dfa26-kube-api-access-hcnkc\") pod \"maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9\" (UID: \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:32.223276 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.223097    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9\" (UID: \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:32.324436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.324404    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9\" (UID: \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:32.324436 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.324439    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcnkc\" (UniqueName: \"kubernetes.io/projected/e9f21b3e-2cae-46db-9e45-58530e9dfa26-kube-api-access-hcnkc\") pod \"maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9\" (UID: \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:32.324660 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.324470    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9\" (UID: \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:32.325236 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.325216    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9\" (UID: \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:32.326935 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.326914    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9\" (UID: \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:32.337143 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.337110    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcnkc\" (UniqueName: \"kubernetes.io/projected/e9f21b3e-2cae-46db-9e45-58530e9dfa26-kube-api-access-hcnkc\") pod \"maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9\" (UID: \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:32.428995 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.428892    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:32.707745 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:32.707716    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"]
Oct 31 12:59:32.709440 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:59:32.709414    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f21b3e_2cae_46db_9e45_58530e9dfa26.slice/crio-539d90628df1f96aff654d0cbaa17571eb0b6b8789e8a609efe7949a732f7d78 WatchSource:0}: Error finding container 539d90628df1f96aff654d0cbaa17571eb0b6b8789e8a609efe7949a732f7d78: Status 404 returned error can't find the container with id 539d90628df1f96aff654d0cbaa17571eb0b6b8789e8a609efe7949a732f7d78
Oct 31 12:59:33.116937 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:33.116886    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c" podUID="01469039-f55a-4ea0-ac8d-e5c8b9a3daaa" containerName="payload-processing" containerID="cri-o://023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98" gracePeriod=25
Oct 31 12:59:33.133041 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:33.133006    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh" podUID="599d803e-9606-46d0-a5bd-d7b5442e1dc2" containerName="payload-pre-processing" containerID="cri-o://7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c" gracePeriod=25
Oct 31 12:59:33.676472 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:33.676439    5663 generic.go:356] "Generic (PLEG): container finished" podID="e9f21b3e-2cae-46db-9e45-58530e9dfa26" containerID="1de6b3211da58618a43160f58bae1ecab6a9f2806ace12090c93451c8141807e" exitCode=0
Oct 31 12:59:33.676662 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:33.676530    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9" event={"ID":"e9f21b3e-2cae-46db-9e45-58530e9dfa26","Type":"ContainerDied","Data":"1de6b3211da58618a43160f58bae1ecab6a9f2806ace12090c93451c8141807e"}
Oct 31 12:59:33.676662 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:33.676595    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9" event={"ID":"e9f21b3e-2cae-46db-9e45-58530e9dfa26","Type":"ContainerStarted","Data":"539d90628df1f96aff654d0cbaa17571eb0b6b8789e8a609efe7949a732f7d78"}
Oct 31 12:59:34.937106 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:34.937078    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:35.080754 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:35.080718    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-openshift-ca\") pod \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\" (UID: \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\") "
Oct 31 12:59:35.080907 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:35.080824    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f21b3e-2cae-46db-9e45-58530e9dfa26-kube-api-access-hcnkc\" (UniqueName: \"kubernetes.io/projected/e9f21b3e-2cae-46db-9e45-58530e9dfa26-kube-api-access-hcnkc\") pod \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\" (UID: \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\") "
Oct 31 12:59:35.080907 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:35.080893    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-xks-ca\") pod \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\" (UID: \"e9f21b3e-2cae-46db-9e45-58530e9dfa26\") "
Oct 31 12:59:35.081089 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:35.081067    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-openshift-ca" pod "e9f21b3e-2cae-46db-9e45-58530e9dfa26" (UID: "e9f21b3e-2cae-46db-9e45-58530e9dfa26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:35.081297 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:35.081273    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:35.082984 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:35.082959    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f21b3e-2cae-46db-9e45-58530e9dfa26-kube-api-access-hcnkc" pod "e9f21b3e-2cae-46db-9e45-58530e9dfa26" (UID: "e9f21b3e-2cae-46db-9e45-58530e9dfa26"). InnerVolumeSpecName "kube-api-access-hcnkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:35.083100 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:35.083007    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-xks-ca" pod "e9f21b3e-2cae-46db-9e45-58530e9dfa26" (UID: "e9f21b3e-2cae-46db-9e45-58530e9dfa26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:35.182965 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:35.182924    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f21b3e-2cae-46db-9e45-58530e9dfa26-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:35.182965 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:35.182958    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcnkc\" (UniqueName: \"kubernetes.io/projected/e9f21b3e-2cae-46db-9e45-58530e9dfa26-kube-api-access-hcnkc\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:35.718840 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:35.718754    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"
Oct 31 12:59:35.718840 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:35.718800    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9" event={"ID":"e9f21b3e-2cae-46db-9e45-58530e9dfa26","Type":"ContainerDied","Data":"539d90628df1f96aff654d0cbaa17571eb0b6b8789e8a609efe7949a732f7d78"}
Oct 31 12:59:35.719007 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:35.718870    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="539d90628df1f96aff654d0cbaa17571eb0b6b8789e8a609efe7949a732f7d78"
Oct 31 12:59:36.983597 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:36.983564    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"]
Oct 31 12:59:37.047330 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:37.047296    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"]
Oct 31 12:59:40.446464 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.446424    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt" podUID="aa472831-ecc0-4dbb-9bcf-4aec35346832" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:40.446953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.446802    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c" podUID="01469039-f55a-4ea0-ac8d-e5c8b9a3daaa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:40.446953 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.446874    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh" podUID="599d803e-9606-46d0-a5bd-d7b5442e1dc2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:40.461631 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.461597    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"]
Oct 31 12:59:40.461780 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.461762    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8" podUID="ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" containerName="discovery" containerID="cri-o://f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff" gracePeriod=30
Oct 31 12:59:40.531160 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.531130    5663 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5jjk8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:8080/ready\": dial tcp 10.133.0.81:8080: connect: connection refused" start-of-body=
Oct 31 12:59:40.531362 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.531209    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8" podUID="ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.81:8080/ready\": dial tcp 10.133.0.81:8080: connect: connection refused"
Oct 31 12:59:40.722698 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.722672    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:59:40.750230 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.750193    5663 generic.go:356] "Generic (PLEG): container finished" podID="ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" containerID="f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff" exitCode=0
Oct 31 12:59:40.750437 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.750319    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8" event={"ID":"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6","Type":"ContainerDied","Data":"f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff"}
Oct 31 12:59:40.750511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.750464    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8" event={"ID":"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6","Type":"ContainerDied","Data":"98028c1def929ece42368160fcedb51244f28fd1ad1acd6d4b22a1a5ea275d07"}
Oct 31 12:59:40.750511 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.750500    5663 scope.go:122] "RemoveContainer" containerID="f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff"
Oct 31 12:59:40.750729 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.750706    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"
Oct 31 12:59:40.760966 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.760928    5663 scope.go:122] "RemoveContainer" containerID="f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff"
Oct 31 12:59:40.761307 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:40.761284    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff\": container with ID starting with f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff not found: ID does not exist" containerID="f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff"
Oct 31 12:59:40.761429 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.761313    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff"} err="failed to get container status \"f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff\": rpc error: code = NotFound desc = could not find container \"f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff\": container with ID starting with f065a86787d0a29f5f7d2d80494331c9fd36721128cf9d2a35c4a704a26e81ff not found: ID does not exist"
Oct 31 12:59:40.769392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.769332    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-cacerts\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-cacerts\") pod \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") "
Oct 31 12:59:40.769487 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.769473    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-local-certs\") pod \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") "
Oct 31 12:59:40.769553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.769539    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-token\" (UniqueName: \"kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-token\") pod \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") "
Oct 31 12:59:40.769615 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.769570    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-kube-api-access-2phz7\" (UniqueName: \"kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-kube-api-access-2phz7\") pod \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") "
Oct 31 12:59:40.769679 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.769659    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-ca-configmap\") pod \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") "
Oct 31 12:59:40.769746 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.769715    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-dns-cert\") pod \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") "
Oct 31 12:59:40.769746 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.769741    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-kubeconfig\") pod \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\" (UID: \"ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6\") "
Oct 31 12:59:40.770418 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.770390    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-ca-configmap" pod "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" (UID: "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:40.772773 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.772749    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-token" pod "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" (UID: "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:40.772887 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.772826    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-kubeconfig" pod "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" (UID: "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:40.773019 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.773000    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-kube-api-access-2phz7" pod "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" (UID: "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6"). InnerVolumeSpecName "kube-api-access-2phz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:40.774132 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.774046    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-cacerts" pod "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" (UID: "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:40.774277 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.774192    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-dns-cert" pod "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" (UID: "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:40.776072 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.776043    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-local-certs" pod "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" (UID: "ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:40.870947 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.870913    5663 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-ca-configmap\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:40.870947 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.870944    5663 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-csr-dns-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:40.870947 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.870955    5663 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-kubeconfig\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:40.871173 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.870965    5663 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-cacerts\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:40.871173 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.870973    5663 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-local-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:40.871173 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.870981    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:40.871173 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:40.870991    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2phz7\" (UniqueName: \"kubernetes.io/projected/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6-kube-api-access-2phz7\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:41.109842 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:41.109809    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"]
Oct 31 12:59:41.114912 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:41.114881    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jjk8"]
Oct 31 12:59:41.438704 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:41.438626    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" path="/var/lib/kubelet/pods/ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6/volumes"
Oct 31 12:59:42.280065 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:42.280023    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"]
Oct 31 12:59:43.135047 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.134989    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"
Oct 31 12:59:43.202760 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.202730    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa472831-ecc0-4dbb-9bcf-4aec35346832-kube-api-access-9mdlz\" (UniqueName: \"kubernetes.io/projected/aa472831-ecc0-4dbb-9bcf-4aec35346832-kube-api-access-9mdlz\") pod \"aa472831-ecc0-4dbb-9bcf-4aec35346832\" (UID: \"aa472831-ecc0-4dbb-9bcf-4aec35346832\") "
Oct 31 12:59:43.202935 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.202918    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa472831-ecc0-4dbb-9bcf-4aec35346832-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa472831-ecc0-4dbb-9bcf-4aec35346832-plugins-config-volume\") pod \"aa472831-ecc0-4dbb-9bcf-4aec35346832\" (UID: \"aa472831-ecc0-4dbb-9bcf-4aec35346832\") "
Oct 31 12:59:43.203277 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.203252    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa472831-ecc0-4dbb-9bcf-4aec35346832-plugins-config-volume" pod "aa472831-ecc0-4dbb-9bcf-4aec35346832" (UID: "aa472831-ecc0-4dbb-9bcf-4aec35346832"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:43.204970 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.204948    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa472831-ecc0-4dbb-9bcf-4aec35346832-kube-api-access-9mdlz" pod "aa472831-ecc0-4dbb-9bcf-4aec35346832" (UID: "aa472831-ecc0-4dbb-9bcf-4aec35346832"). InnerVolumeSpecName "kube-api-access-9mdlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:43.304556 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.304524    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa472831-ecc0-4dbb-9bcf-4aec35346832-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:43.304556 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.304555    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mdlz\" (UniqueName: \"kubernetes.io/projected/aa472831-ecc0-4dbb-9bcf-4aec35346832-kube-api-access-9mdlz\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:43.777068 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.776978    5663 generic.go:356] "Generic (PLEG): container finished" podID="aa472831-ecc0-4dbb-9bcf-4aec35346832" containerID="13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180" exitCode=0
Oct 31 12:59:43.777068 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.777039    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"
Oct 31 12:59:43.777292 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.777061    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt" event={"ID":"aa472831-ecc0-4dbb-9bcf-4aec35346832","Type":"ContainerDied","Data":"13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180"}
Oct 31 12:59:43.777292 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.777166    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt" event={"ID":"aa472831-ecc0-4dbb-9bcf-4aec35346832","Type":"ContainerDied","Data":"02c1ef274c08a014c8aeeb04fb23103012a31baa0070a48199d7dade3ee20e13"}
Oct 31 12:59:43.777292 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.777199    5663 scope.go:122] "RemoveContainer" containerID="13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180"
Oct 31 12:59:43.786998 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.786974    5663 scope.go:122] "RemoveContainer" containerID="13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180"
Oct 31 12:59:43.787256 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:43.787236    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180\": container with ID starting with 13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180 not found: ID does not exist" containerID="13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180"
Oct 31 12:59:43.787306 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.787264    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180"} err="failed to get container status \"13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180\": rpc error: code = NotFound desc = could not find container \"13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180\": container with ID starting with 13fb6c3a0ef9bec2a3c803b4b9900562eb47a18605dc03356643a82316af4180 not found: ID does not exist"
Oct 31 12:59:43.796318 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.796294    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"]
Oct 31 12:59:43.800903 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:43.800880    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2c393f-7898d69bdc-hmhmt"]
Oct 31 12:59:44.807696 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.807665    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"]
Oct 31 12:59:44.808149 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.808119    5663 cpu_manager.go:414] "containerMap: removing container" podUID="ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" containerName="discovery"
Oct 31 12:59:44.808149 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.808147    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" containerName="discovery"
Oct 31 12:59:44.808420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.808163    5663 cpu_manager.go:414] "containerMap: removing container" podUID="e9f21b3e-2cae-46db-9e45-58530e9dfa26" containerName="revoke-keys"
Oct 31 12:59:44.808420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.808171    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f21b3e-2cae-46db-9e45-58530e9dfa26" containerName="revoke-keys"
Oct 31 12:59:44.808420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.808189    5663 cpu_manager.go:414] "containerMap: removing container" podUID="aa472831-ecc0-4dbb-9bcf-4aec35346832" containerName="payload-pre-processing"
Oct 31 12:59:44.808420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.808196    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa472831-ecc0-4dbb-9bcf-4aec35346832" containerName="payload-pre-processing"
Oct 31 12:59:44.808420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.808318    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae87d4b1-ee30-4565-9c7d-1e8ecf251cc6" containerName="discovery"
Oct 31 12:59:44.808420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.808327    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f21b3e-2cae-46db-9e45-58530e9dfa26" containerName="revoke-keys"
Oct 31 12:59:44.808420 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.808334    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa472831-ecc0-4dbb-9bcf-4aec35346832" containerName="payload-pre-processing"
Oct 31 12:59:44.811657 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.811638    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:44.819432 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.819411    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 12:59:44.820650 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.820611    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 12:59:44.824275 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.824240    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"]
Oct 31 12:59:44.835970 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.835934    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8w9b\" (UniqueName: \"kubernetes.io/projected/d69e066f-edff-4419-8975-bac11fe1a8f3-kube-api-access-x8w9b\") pod \"maas-api-revoke-sub-ce73be33-pgw6k\" (UID: \"d69e066f-edff-4419-8975-bac11fe1a8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:44.836133 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.836004    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce73be33-pgw6k\" (UID: \"d69e066f-edff-4419-8975-bac11fe1a8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:44.836206 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.836179    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce73be33-pgw6k\" (UID: \"d69e066f-edff-4419-8975-bac11fe1a8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:44.937535 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.937487    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce73be33-pgw6k\" (UID: \"d69e066f-edff-4419-8975-bac11fe1a8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:44.937740 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.937598    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8w9b\" (UniqueName: \"kubernetes.io/projected/d69e066f-edff-4419-8975-bac11fe1a8f3-kube-api-access-x8w9b\") pod \"maas-api-revoke-sub-ce73be33-pgw6k\" (UID: \"d69e066f-edff-4419-8975-bac11fe1a8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:44.937740 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.937646    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce73be33-pgw6k\" (UID: \"d69e066f-edff-4419-8975-bac11fe1a8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:44.938485 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.938458    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce73be33-pgw6k\" (UID: \"d69e066f-edff-4419-8975-bac11fe1a8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:44.939860 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.939836    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce73be33-pgw6k\" (UID: \"d69e066f-edff-4419-8975-bac11fe1a8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:44.949492 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:44.949470    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8w9b\" (UniqueName: \"kubernetes.io/projected/d69e066f-edff-4419-8975-bac11fe1a8f3-kube-api-access-x8w9b\") pod \"maas-api-revoke-sub-ce73be33-pgw6k\" (UID: \"d69e066f-edff-4419-8975-bac11fe1a8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:45.135434 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:45.129092    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:45.422388 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:45.422345    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"]
Oct 31 12:59:45.424178 ip-10-0-140-137 kubenswrapper[5663]: W1031 12:59:45.424143    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69e066f_edff_4419_8975_bac11fe1a8f3.slice/crio-d299db1beabb516b47347f7ab563ff3126240743f4190e14cb1d6f79741e03b6 WatchSource:0}: Error finding container d299db1beabb516b47347f7ab563ff3126240743f4190e14cb1d6f79741e03b6: Status 404 returned error can't find the container with id d299db1beabb516b47347f7ab563ff3126240743f4190e14cb1d6f79741e03b6
Oct 31 12:59:45.438819 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:45.438799    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa472831-ecc0-4dbb-9bcf-4aec35346832" path="/var/lib/kubelet/pods/aa472831-ecc0-4dbb-9bcf-4aec35346832/volumes"
Oct 31 12:59:45.798297 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:45.798254    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k" event={"ID":"d69e066f-edff-4419-8975-bac11fe1a8f3","Type":"ContainerStarted","Data":"9ed9d8391d488b4e777a447247aa5d4c9b474810ebe6fd0bb93399c743185282"}
Oct 31 12:59:45.798513 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:45.798400    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k" event={"ID":"d69e066f-edff-4419-8975-bac11fe1a8f3","Type":"ContainerStarted","Data":"d299db1beabb516b47347f7ab563ff3126240743f4190e14cb1d6f79741e03b6"}
Oct 31 12:59:45.819515 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:45.819467    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k" podStartSLOduration=1.819453195 podStartE2EDuration="1.819453195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:45.81632594 +0000 UTC m=+1910.556752150" watchObservedRunningTime="2026-10-31 12:59:45.819453195 +0000 UTC m=+1910.559879458"
Oct 31 12:59:46.807137 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:46.807101    5663 generic.go:356] "Generic (PLEG): container finished" podID="d69e066f-edff-4419-8975-bac11fe1a8f3" containerID="9ed9d8391d488b4e777a447247aa5d4c9b474810ebe6fd0bb93399c743185282" exitCode=0
Oct 31 12:59:46.807288 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:46.807190    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k" event={"ID":"d69e066f-edff-4419-8975-bac11fe1a8f3","Type":"ContainerDied","Data":"9ed9d8391d488b4e777a447247aa5d4c9b474810ebe6fd0bb93399c743185282"}
Oct 31 12:59:46.930935 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:46.930904    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"]
Oct 31 12:59:46.931335 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:46.931122    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy" containerID="cri-o://b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e" gracePeriod=30
Oct 31 12:59:47.281490 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:47.281404    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x" podUID="d279e306-b346-462a-a120-9fb77b6dcb0b" containerName="payload-pre-processing" containerID="cri-o://68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3" gracePeriod=25
Oct 31 12:59:47.990345 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:47.990322    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:48.095033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:48.094957    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69e066f-edff-4419-8975-bac11fe1a8f3-kube-api-access-x8w9b\" (UniqueName: \"kubernetes.io/projected/d69e066f-edff-4419-8975-bac11fe1a8f3-kube-api-access-x8w9b\") pod \"d69e066f-edff-4419-8975-bac11fe1a8f3\" (UID: \"d69e066f-edff-4419-8975-bac11fe1a8f3\") "
Oct 31 12:59:48.095033 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:48.095013    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-openshift-ca\") pod \"d69e066f-edff-4419-8975-bac11fe1a8f3\" (UID: \"d69e066f-edff-4419-8975-bac11fe1a8f3\") "
Oct 31 12:59:48.095240 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:48.095126    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-xks-ca\") pod \"d69e066f-edff-4419-8975-bac11fe1a8f3\" (UID: \"d69e066f-edff-4419-8975-bac11fe1a8f3\") "
Oct 31 12:59:48.095413 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:48.095388    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-openshift-ca" pod "d69e066f-edff-4419-8975-bac11fe1a8f3" (UID: "d69e066f-edff-4419-8975-bac11fe1a8f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:48.097204 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:48.097179    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-xks-ca" pod "d69e066f-edff-4419-8975-bac11fe1a8f3" (UID: "d69e066f-edff-4419-8975-bac11fe1a8f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:48.097314 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:48.097244    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69e066f-edff-4419-8975-bac11fe1a8f3-kube-api-access-x8w9b" pod "d69e066f-edff-4419-8975-bac11fe1a8f3" (UID: "d69e066f-edff-4419-8975-bac11fe1a8f3"). InnerVolumeSpecName "kube-api-access-x8w9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:48.196614 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:48.196584    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8w9b\" (UniqueName: \"kubernetes.io/projected/d69e066f-edff-4419-8975-bac11fe1a8f3-kube-api-access-x8w9b\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.196614 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:48.196612    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.196812 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:48.196623    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d69e066f-edff-4419-8975-bac11fe1a8f3-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:48.822250 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:48.822218    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k" event={"ID":"d69e066f-edff-4419-8975-bac11fe1a8f3","Type":"ContainerDied","Data":"d299db1beabb516b47347f7ab563ff3126240743f4190e14cb1d6f79741e03b6"}
Oct 31 12:59:48.833534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:48.822261    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"
Oct 31 12:59:48.833534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:48.822269    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d299db1beabb516b47347f7ab563ff3126240743f4190e14cb1d6f79741e03b6"
Oct 31 12:59:50.446605 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:50.446566    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh" podUID="599d803e-9606-46d0-a5bd-d7b5442e1dc2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:50.446989 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:50.446684    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c" podUID="01469039-f55a-4ea0-ac8d-e5c8b9a3daaa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:52.003758 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.003703    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" containerID="cri-o://c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a" gracePeriod=45
Oct 31 12:59:52.050610 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.050578    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": dial tcp 10.133.0.49:8000: connect: connection refused"
Oct 31 12:59:52.068739 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.068679    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" containerID="cri-o://e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74" gracePeriod=45
Oct 31 12:59:52.170265 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.170049    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": dial tcp 10.133.0.39:8000: connect: connection refused"
Oct 31 12:59:52.265499 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.265464    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:59:52.324311 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.323975    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:59:52.360327 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.360291    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:59:52.361554 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.361525    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-kube-api-access-c79g4\" (UniqueName: \"kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-kube-api-access-c79g4\") pod \"353c883f-02b3-49e1-83bf-e874321bfcca\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") "
Oct 31 12:59:52.361733 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.361576    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-istio-token\" (UniqueName: \"kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-istio-token\") pod \"353c883f-02b3-49e1-83bf-e874321bfcca\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") "
Oct 31 12:59:52.361733 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.361622    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-credential-socket\") pod \"353c883f-02b3-49e1-83bf-e874321bfcca\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") "
Oct 31 12:59:52.361733 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.361709    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-envoy\") pod \"353c883f-02b3-49e1-83bf-e874321bfcca\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") "
Oct 31 12:59:52.361908 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.361738    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-socket\") pod \"353c883f-02b3-49e1-83bf-e874321bfcca\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") "
Oct 31 12:59:52.361908 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.361791    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-data\") pod \"353c883f-02b3-49e1-83bf-e874321bfcca\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") "
Oct 31 12:59:52.362023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.361924    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353c883f-02b3-49e1-83bf-e874321bfcca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353c883f-02b3-49e1-83bf-e874321bfcca-istiod-ca-cert\") pod \"353c883f-02b3-49e1-83bf-e874321bfcca\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") "
Oct 31 12:59:52.362023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.361967    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-certs\") pod \"353c883f-02b3-49e1-83bf-e874321bfcca\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") "
Oct 31 12:59:52.362023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.361988    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/353c883f-02b3-49e1-83bf-e874321bfcca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353c883f-02b3-49e1-83bf-e874321bfcca-istio-podinfo\") pod \"353c883f-02b3-49e1-83bf-e874321bfcca\" (UID: \"353c883f-02b3-49e1-83bf-e874321bfcca\") "
Oct 31 12:59:52.363809 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.363487    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-socket" pod "353c883f-02b3-49e1-83bf-e874321bfcca" (UID: "353c883f-02b3-49e1-83bf-e874321bfcca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.363809 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.363590    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-data" pod "353c883f-02b3-49e1-83bf-e874321bfcca" (UID: "353c883f-02b3-49e1-83bf-e874321bfcca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.363809 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.363699    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-certs" pod "353c883f-02b3-49e1-83bf-e874321bfcca" (UID: "353c883f-02b3-49e1-83bf-e874321bfcca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.363809 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.363777    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353c883f-02b3-49e1-83bf-e874321bfcca-istiod-ca-cert" pod "353c883f-02b3-49e1-83bf-e874321bfcca" (UID: "353c883f-02b3-49e1-83bf-e874321bfcca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:52.365157 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.364437    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-credential-socket" pod "353c883f-02b3-49e1-83bf-e874321bfcca" (UID: "353c883f-02b3-49e1-83bf-e874321bfcca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.365310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.365144    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-istio-token" pod "353c883f-02b3-49e1-83bf-e874321bfcca" (UID: "353c883f-02b3-49e1-83bf-e874321bfcca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:52.365310 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.365191    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-envoy" pod "353c883f-02b3-49e1-83bf-e874321bfcca" (UID: "353c883f-02b3-49e1-83bf-e874321bfcca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.365630 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.365607    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/353c883f-02b3-49e1-83bf-e874321bfcca-istio-podinfo" pod "353c883f-02b3-49e1-83bf-e874321bfcca" (UID: "353c883f-02b3-49e1-83bf-e874321bfcca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:52.369245 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.369220    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-kube-api-access-c79g4" pod "353c883f-02b3-49e1-83bf-e874321bfcca" (UID: "353c883f-02b3-49e1-83bf-e874321bfcca"). InnerVolumeSpecName "kube-api-access-c79g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:52.463278 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463242    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-home\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-home\") pod \"e69182cf-2356-4273-84be-13030f3405cd\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") "
Oct 31 12:59:52.463501 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463386    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-tmp-dir\") pod \"e69182cf-2356-4273-84be-13030f3405cd\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") "
Oct 31 12:59:52.463501 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463414    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-model-cache\") pod \"e69182cf-2356-4273-84be-13030f3405cd\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") "
Oct 31 12:59:52.463501 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463442    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-dshm\") pod \"e69182cf-2356-4273-84be-13030f3405cd\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") "
Oct 31 12:59:52.463501 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463495    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69182cf-2356-4273-84be-13030f3405cd-kube-api-access-4njvt\" (UniqueName: \"kubernetes.io/projected/e69182cf-2356-4273-84be-13030f3405cd-kube-api-access-4njvt\") pod \"e69182cf-2356-4273-84be-13030f3405cd\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") "
Oct 31 12:59:52.463759 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463546    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tmp-dir\") pod \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") "
Oct 31 12:59:52.463759 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463577    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-model-cache\") pod \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") "
Oct 31 12:59:52.463759 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463606    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69182cf-2356-4273-84be-13030f3405cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/e69182cf-2356-4273-84be-13030f3405cd-tls-certs\") pod \"e69182cf-2356-4273-84be-13030f3405cd\" (UID: \"e69182cf-2356-4273-84be-13030f3405cd\") "
Oct 31 12:59:52.463759 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463621    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-home" pod "e69182cf-2356-4273-84be-13030f3405cd" (UID: "e69182cf-2356-4273-84be-13030f3405cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.463759 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463649    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tls-certs\") pod \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") "
Oct 31 12:59:52.463759 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463686    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-dshm\") pod \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") "
Oct 31 12:59:52.464195 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463762    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-home\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-home\") pod \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") "
Oct 31 12:59:52.464195 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463801    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-kube-api-access-b8wqn\" (UniqueName: \"kubernetes.io/projected/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-kube-api-access-b8wqn\") pod \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\" (UID: \"a6546f0e-5df8-44d9-9ed1-d7a115a6441a\") "
Oct 31 12:59:52.464195 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.463885    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-model-cache" pod "a6546f0e-5df8-44d9-9ed1-d7a115a6441a" (UID: "a6546f0e-5df8-44d9-9ed1-d7a115a6441a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.464195 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464111    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tmp-dir" pod "a6546f0e-5df8-44d9-9ed1-d7a115a6441a" (UID: "a6546f0e-5df8-44d9-9ed1-d7a115a6441a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.464579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464287    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.464579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464311    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.464579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464327    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/353c883f-02b3-49e1-83bf-e874321bfcca-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.464579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464342    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.464579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464373    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/353c883f-02b3-49e1-83bf-e874321bfcca-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.464579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464387    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c79g4\" (UniqueName: \"kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-kube-api-access-c79g4\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.464579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464403    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/353c883f-02b3-49e1-83bf-e874321bfcca-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.464579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464416    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.464579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464429    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.464579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464442    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.464579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464458    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.464579 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464470    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/353c883f-02b3-49e1-83bf-e874321bfcca-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.465272 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.464704    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-home" pod "a6546f0e-5df8-44d9-9ed1-d7a115a6441a" (UID: "a6546f0e-5df8-44d9-9ed1-d7a115a6441a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.467024 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.466968    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-dshm" pod "a6546f0e-5df8-44d9-9ed1-d7a115a6441a" (UID: "a6546f0e-5df8-44d9-9ed1-d7a115a6441a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.467732 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.467686    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-tmp-dir" pod "e69182cf-2356-4273-84be-13030f3405cd" (UID: "e69182cf-2356-4273-84be-13030f3405cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.470148 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.470118    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-dshm" pod "e69182cf-2356-4273-84be-13030f3405cd" (UID: "e69182cf-2356-4273-84be-13030f3405cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.470148 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.470140    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-kube-api-access-b8wqn" pod "a6546f0e-5df8-44d9-9ed1-d7a115a6441a" (UID: "a6546f0e-5df8-44d9-9ed1-d7a115a6441a"). InnerVolumeSpecName "kube-api-access-b8wqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:52.470148 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.470137    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69182cf-2356-4273-84be-13030f3405cd-tls-certs" pod "e69182cf-2356-4273-84be-13030f3405cd" (UID: "e69182cf-2356-4273-84be-13030f3405cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:52.470412 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.470122    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69182cf-2356-4273-84be-13030f3405cd-kube-api-access-4njvt" pod "e69182cf-2356-4273-84be-13030f3405cd" (UID: "e69182cf-2356-4273-84be-13030f3405cd"). InnerVolumeSpecName "kube-api-access-4njvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:52.470412 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.470207    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-model-cache" pod "e69182cf-2356-4273-84be-13030f3405cd" (UID: "e69182cf-2356-4273-84be-13030f3405cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:52.471023 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.470999    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tls-certs" pod "a6546f0e-5df8-44d9-9ed1-d7a115a6441a" (UID: "a6546f0e-5df8-44d9-9ed1-d7a115a6441a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 12:59:52.483926 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:52.483831    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 12:59:52.565326 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.565295    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.565326 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.565324    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.565553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.565335    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69182cf-2356-4273-84be-13030f3405cd-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.565553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.565345    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4njvt\" (UniqueName: \"kubernetes.io/projected/e69182cf-2356-4273-84be-13030f3405cd-kube-api-access-4njvt\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.565553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.565386    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69182cf-2356-4273-84be-13030f3405cd-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.565553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.565394    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.565553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.565403    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.565553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.565411    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.565553 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.565420    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8wqn\" (UniqueName: \"kubernetes.io/projected/a6546f0e-5df8-44d9-9ed1-d7a115a6441a-kube-api-access-b8wqn\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:52.847082 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.847046    5663 generic.go:356] "Generic (PLEG): container finished" podID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerID="e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74" exitCode=0
Oct 31 12:59:52.847292 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.847094    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"
Oct 31 12:59:52.847292 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.847133    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" event={"ID":"a6546f0e-5df8-44d9-9ed1-d7a115a6441a","Type":"ContainerDied","Data":"e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74"}
Oct 31 12:59:52.847292 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.847182    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr" event={"ID":"a6546f0e-5df8-44d9-9ed1-d7a115a6441a","Type":"ContainerDied","Data":"70e3dba285f80c9a0b7f010befd1f7492646d6eed0f64d20d14d43c196b0b23b"}
Oct 31 12:59:52.847292 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.847199    5663 scope.go:122] "RemoveContainer" containerID="e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74"
Oct 31 12:59:52.853399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.849063    5663 generic.go:356] "Generic (PLEG): container finished" podID="353c883f-02b3-49e1-83bf-e874321bfcca" containerID="b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e" exitCode=0
Oct 31 12:59:52.853399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.849107    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" event={"ID":"353c883f-02b3-49e1-83bf-e874321bfcca","Type":"ContainerDied","Data":"b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e"}
Oct 31 12:59:52.853399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.849156    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp" event={"ID":"353c883f-02b3-49e1-83bf-e874321bfcca","Type":"ContainerDied","Data":"77e2f0627d8a1f851887bb7266355614a91de9953160c3da616079b3a4a28de1"}
Oct 31 12:59:52.853399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.849113    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"
Oct 31 12:59:52.853399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.850773    5663 generic.go:356] "Generic (PLEG): container finished" podID="e69182cf-2356-4273-84be-13030f3405cd" containerID="c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a" exitCode=0
Oct 31 12:59:52.853399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.850809    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"
Oct 31 12:59:52.853399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.850832    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" event={"ID":"e69182cf-2356-4273-84be-13030f3405cd","Type":"ContainerDied","Data":"c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a"}
Oct 31 12:59:52.853399 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.850861    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt" event={"ID":"e69182cf-2356-4273-84be-13030f3405cd","Type":"ContainerDied","Data":"3df5ef2fbd331c218626f6036feee6db192997b0f185848c709322711898ed8f"}
Oct 31 12:59:52.859079 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.859051    5663 scope.go:122] "RemoveContainer" containerID="e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74"
Oct 31 12:59:52.859423 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:52.859400    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74\": container with ID starting with e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74 not found: ID does not exist" containerID="e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74"
Oct 31 12:59:52.859513 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.859432    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74"} err="failed to get container status \"e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74\": rpc error: code = NotFound desc = could not find container \"e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74\": container with ID starting with e86f07bbee7df0dc2861fa84a1bc0ea0f63bc4bca2eb8e9edaff4e8e6f497e74 not found: ID does not exist"
Oct 31 12:59:52.859513 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.859450    5663 scope.go:122] "RemoveContainer" containerID="b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e"
Oct 31 12:59:52.869423 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.869407    5663 scope.go:122] "RemoveContainer" containerID="b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e"
Oct 31 12:59:52.869688 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:52.869669    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e\": container with ID starting with b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e not found: ID does not exist" containerID="b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e"
Oct 31 12:59:52.869774 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.869698    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e"} err="failed to get container status \"b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e\": rpc error: code = NotFound desc = could not find container \"b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e\": container with ID starting with b8e180ba7a37eee34d9a84088b6a4a808b96bcab9ea8d98bf6644f6c54810a1e not found: ID does not exist"
Oct 31 12:59:52.869774 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.869721    5663 scope.go:122] "RemoveContainer" containerID="c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a"
Oct 31 12:59:52.878186 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.878161    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"]
Oct 31 12:59:52.879873 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.879855    5663 scope.go:122] "RemoveContainer" containerID="c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a"
Oct 31 12:59:52.880189 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:52.880168    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a\": container with ID starting with c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a not found: ID does not exist" containerID="c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a"
Oct 31 12:59:52.880281 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.880195    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a"} err="failed to get container status \"c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a\": rpc error: code = NotFound desc = could not find container \"c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a\": container with ID starting with c2331186144a0eaf43b9211f3d548b7567fb52ec317af982ea605272a2ee492a not found: ID does not exist"
Oct 31 12:59:52.880862 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.880844    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-ms7fp"]
Oct 31 12:59:52.892216 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.892193    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"]
Oct 31 12:59:52.898888 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.898865    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/simulator-w5-b25d3b-kserve-6cc7b5c596-vmtjr"]
Oct 31 12:59:52.910052 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.910023    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"]
Oct 31 12:59:52.918392 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:52.918364    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b25d3b/distinct-w5-b25d3b-kserve-6cf7c484cb-bnfbt"]
Oct 31 12:59:53.442626 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.442594    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" path="/var/lib/kubelet/pods/353c883f-02b3-49e1-83bf-e874321bfcca/volumes"
Oct 31 12:59:53.443168 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.443148    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" path="/var/lib/kubelet/pods/a6546f0e-5df8-44d9-9ed1-d7a115a6441a/volumes"
Oct 31 12:59:53.443659 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.443643    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69182cf-2356-4273-84be-13030f3405cd" path="/var/lib/kubelet/pods/e69182cf-2356-4273-84be-13030f3405cd/volumes"
Oct 31 12:59:53.462993 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.458263    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"
Oct 31 12:59:53.525811 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.525778    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"
Oct 31 12:59:53.583768 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.583736    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-plugins-config-volume\") pod \"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa\" (UID: \"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa\") "
Oct 31 12:59:53.583927 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.583903    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-kube-api-access-cfbvt\" (UniqueName: \"kubernetes.io/projected/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-kube-api-access-cfbvt\") pod \"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa\" (UID: \"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa\") "
Oct 31 12:59:53.584194 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.584169    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-plugins-config-volume" pod "01469039-f55a-4ea0-ac8d-e5c8b9a3daaa" (UID: "01469039-f55a-4ea0-ac8d-e5c8b9a3daaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:53.586051 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.586027    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-kube-api-access-cfbvt" pod "01469039-f55a-4ea0-ac8d-e5c8b9a3daaa" (UID: "01469039-f55a-4ea0-ac8d-e5c8b9a3daaa"). InnerVolumeSpecName "kube-api-access-cfbvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:53.686315 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.686224    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/599d803e-9606-46d0-a5bd-d7b5442e1dc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599d803e-9606-46d0-a5bd-d7b5442e1dc2-plugins-config-volume\") pod \"599d803e-9606-46d0-a5bd-d7b5442e1dc2\" (UID: \"599d803e-9606-46d0-a5bd-d7b5442e1dc2\") "
Oct 31 12:59:53.686653 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.686631    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599d803e-9606-46d0-a5bd-d7b5442e1dc2-kube-api-access-ww7w6\" (UniqueName: \"kubernetes.io/projected/599d803e-9606-46d0-a5bd-d7b5442e1dc2-kube-api-access-ww7w6\") pod \"599d803e-9606-46d0-a5bd-d7b5442e1dc2\" (UID: \"599d803e-9606-46d0-a5bd-d7b5442e1dc2\") "
Oct 31 12:59:53.687785 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.687759    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:53.687921 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.687909    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfbvt\" (UniqueName: \"kubernetes.io/projected/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa-kube-api-access-cfbvt\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:53.688771 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.688745    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599d803e-9606-46d0-a5bd-d7b5442e1dc2-plugins-config-volume" pod "599d803e-9606-46d0-a5bd-d7b5442e1dc2" (UID: "599d803e-9606-46d0-a5bd-d7b5442e1dc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:53.691326 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.691299    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599d803e-9606-46d0-a5bd-d7b5442e1dc2-kube-api-access-ww7w6" pod "599d803e-9606-46d0-a5bd-d7b5442e1dc2" (UID: "599d803e-9606-46d0-a5bd-d7b5442e1dc2"). InnerVolumeSpecName "kube-api-access-ww7w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:53.789330 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.789295    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599d803e-9606-46d0-a5bd-d7b5442e1dc2-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:53.789330 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.789327    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww7w6\" (UniqueName: \"kubernetes.io/projected/599d803e-9606-46d0-a5bd-d7b5442e1dc2-kube-api-access-ww7w6\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:53.860285 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.860246    5663 generic.go:356] "Generic (PLEG): container finished" podID="599d803e-9606-46d0-a5bd-d7b5442e1dc2" containerID="7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c" exitCode=0
Oct 31 12:59:53.860460 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.860294    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"
Oct 31 12:59:53.860460 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.860319    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh" event={"ID":"599d803e-9606-46d0-a5bd-d7b5442e1dc2","Type":"ContainerDied","Data":"7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c"}
Oct 31 12:59:53.860460 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.860421    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh" event={"ID":"599d803e-9606-46d0-a5bd-d7b5442e1dc2","Type":"ContainerDied","Data":"fb199aa0db80543799b391242d9c424492d4557c26083b6234bc6d881eb2e650"}
Oct 31 12:59:53.860589 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.860461    5663 scope.go:122] "RemoveContainer" containerID="7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c"
Oct 31 12:59:53.861758 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.861734    5663 generic.go:356] "Generic (PLEG): container finished" podID="01469039-f55a-4ea0-ac8d-e5c8b9a3daaa" containerID="023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98" exitCode=0
Oct 31 12:59:53.861886 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.861771    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c" event={"ID":"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa","Type":"ContainerDied","Data":"023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98"}
Oct 31 12:59:53.861886 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.861831    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c" event={"ID":"01469039-f55a-4ea0-ac8d-e5c8b9a3daaa","Type":"ContainerDied","Data":"637c7e9a7108af67b8dd0737999f4fe1811e5cc7e41af83cfd71614d3cd343ae"}
Oct 31 12:59:53.861886 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.861773    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"
Oct 31 12:59:53.870651 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.870635    5663 scope.go:122] "RemoveContainer" containerID="7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c"
Oct 31 12:59:53.870920 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:53.870893    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c\": container with ID starting with 7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c not found: ID does not exist" containerID="7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c"
Oct 31 12:59:53.871053 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.870928    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c"} err="failed to get container status \"7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c\": rpc error: code = NotFound desc = could not find container \"7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c\": container with ID starting with 7e53a3568875f310cd86de9b80900d0f0f55ddb3dd8c1fd820ae00fa6924684c not found: ID does not exist"
Oct 31 12:59:53.871053 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.870947    5663 scope.go:122] "RemoveContainer" containerID="023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98"
Oct 31 12:59:53.882894 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.882874    5663 scope.go:122] "RemoveContainer" containerID="023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98"
Oct 31 12:59:53.883258 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:53.883231    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98\": container with ID starting with 023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98 not found: ID does not exist" containerID="023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98"
Oct 31 12:59:53.883632 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.883501    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98"} err="failed to get container status \"023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98\": rpc error: code = NotFound desc = could not find container \"023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98\": container with ID starting with 023d3f8eea5cc58d1c9e8d8ea8a95a4fd0b8dcd43b170cc5a75d787ef1ba0c98 not found: ID does not exist"
Oct 31 12:59:53.898530 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.898495    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"]
Oct 31 12:59:53.901320 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.901291    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fd988c-68696fc8d7-rl29c"]
Oct 31 12:59:53.917410 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.917380    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"]
Oct 31 12:59:53.920047 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.920019    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"]
Oct 31 12:59:53.920298 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.920271    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy" containerID="cri-o://94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb" gracePeriod=30
Oct 31 12:59:53.922448 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:53.922418    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fd988c-6964bf57ff-dlvkh"]
Oct 31 12:59:54.758501 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:54.758464    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x" podUID="d279e306-b346-462a-a120-9fb77b6dcb0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 12:59:55.146129 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:55.146096    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"]
Oct 31 12:59:55.146316 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:55.146294    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy" containerID="cri-o://53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1" gracePeriod=30
Oct 31 12:59:55.192056 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:55.192023    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"]
Oct 31 12:59:55.192479 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:55.192442    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerName="istio-proxy" containerID="cri-o://43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a" gracePeriod=30
Oct 31 12:59:55.439094 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:55.439008    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01469039-f55a-4ea0-ac8d-e5c8b9a3daaa" path="/var/lib/kubelet/pods/01469039-f55a-4ea0-ac8d-e5c8b9a3daaa/volumes"
Oct 31 12:59:55.439534 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:55.439514    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599d803e-9606-46d0-a5bd-d7b5442e1dc2" path="/var/lib/kubelet/pods/599d803e-9606-46d0-a5bd-d7b5442e1dc2/volumes"
Oct 31 12:59:58.888417 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:58.888387    5663 patch_prober.go:29] interesting pod/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 12:59:58.888866 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:58.888450    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 12:59:59.145450 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.145216    5663 scope.go:122] "RemoveContainer" containerID="b294badace00abaa3f7263a8c79a13ac5e6fb79325aa7364782d0c047c0a6e90"
Oct 31 12:59:59.158034 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.158009    5663 scope.go:122] "RemoveContainer" containerID="cbec050c1d07b9c5f7c745d6de64289e0d797b2b0db9731ae7566049fed690ce"
Oct 31 12:59:59.205941 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.205917    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.304709    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-envoy\") pod \"342f09b7-245d-4235-84da-8625ac04f790\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") "
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.304768    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-kube-api-access-899pt\" (UniqueName: \"kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-kube-api-access-899pt\") pod \"342f09b7-245d-4235-84da-8625ac04f790\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") "
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.304836    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-certs\") pod \"342f09b7-245d-4235-84da-8625ac04f790\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") "
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.304873    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-credential-socket\") pod \"342f09b7-245d-4235-84da-8625ac04f790\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") "
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.304925    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/342f09b7-245d-4235-84da-8625ac04f790-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342f09b7-245d-4235-84da-8625ac04f790-istio-podinfo\") pod \"342f09b7-245d-4235-84da-8625ac04f790\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") "
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.304980    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-socket\") pod \"342f09b7-245d-4235-84da-8625ac04f790\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") "
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.305027    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-data\") pod \"342f09b7-245d-4235-84da-8625ac04f790\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") "
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.305115    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/342f09b7-245d-4235-84da-8625ac04f790-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342f09b7-245d-4235-84da-8625ac04f790-istiod-ca-cert\") pod \"342f09b7-245d-4235-84da-8625ac04f790\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") "
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.305157    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-istio-token\" (UniqueName: \"kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-istio-token\") pod \"342f09b7-245d-4235-84da-8625ac04f790\" (UID: \"342f09b7-245d-4235-84da-8625ac04f790\") "
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.324294    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-credential-socket" pod "342f09b7-245d-4235-84da-8625ac04f790" (UID: "342f09b7-245d-4235-84da-8625ac04f790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.324811    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-socket" pod "342f09b7-245d-4235-84da-8625ac04f790" (UID: "342f09b7-245d-4235-84da-8625ac04f790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.325302    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-data" pod "342f09b7-245d-4235-84da-8625ac04f790" (UID: "342f09b7-245d-4235-84da-8625ac04f790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.327704    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-certs" pod "342f09b7-245d-4235-84da-8625ac04f790" (UID: "342f09b7-245d-4235-84da-8625ac04f790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.327719    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/342f09b7-245d-4235-84da-8625ac04f790-istio-podinfo" pod "342f09b7-245d-4235-84da-8625ac04f790" (UID: "342f09b7-245d-4235-84da-8625ac04f790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 12:59:59.340390 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.327995    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342f09b7-245d-4235-84da-8625ac04f790-istiod-ca-cert" pod "342f09b7-245d-4235-84da-8625ac04f790" (UID: "342f09b7-245d-4235-84da-8625ac04f790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 12:59:59.355162 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.353129    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-kube-api-access-899pt" pod "342f09b7-245d-4235-84da-8625ac04f790" (UID: "342f09b7-245d-4235-84da-8625ac04f790"). InnerVolumeSpecName "kube-api-access-899pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:59.355162 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.354496    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-envoy" pod "342f09b7-245d-4235-84da-8625ac04f790" (UID: "342f09b7-245d-4235-84da-8625ac04f790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 12:59:59.399875 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.399796    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-istio-token" pod "342f09b7-245d-4235-84da-8625ac04f790" (UID: "342f09b7-245d-4235-84da-8625ac04f790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 12:59:59.407547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.407325    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:59.407547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.407395    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342f09b7-245d-4235-84da-8625ac04f790-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:59.407547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.407411    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:59.407547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.407427    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:59.407547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.407441    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-899pt\" (UniqueName: \"kubernetes.io/projected/342f09b7-245d-4235-84da-8625ac04f790-kube-api-access-899pt\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:59.407547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.407458    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:59.407547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.407472    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:59.407547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.407487    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342f09b7-245d-4235-84da-8625ac04f790-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:59.407547 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.407502    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342f09b7-245d-4235-84da-8625ac04f790-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 12:59:59.429495 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.423082    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"]
Oct 31 12:59:59.429495 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.427977    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b25d3b-9cst9"]
Oct 31 12:59:59.438099 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.438072    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f21b3e-2cae-46db-9e45-58530e9dfa26" path="/var/lib/kubelet/pods/e9f21b3e-2cae-46db-9e45-58530e9dfa26/volumes"
Oct 31 12:59:59.907004 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.906973    5663 generic.go:356] "Generic (PLEG): container finished" podID="342f09b7-245d-4235-84da-8625ac04f790" containerID="94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb" exitCode=0
Oct 31 12:59:59.907446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.907032    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" event={"ID":"342f09b7-245d-4235-84da-8625ac04f790","Type":"ContainerDied","Data":"94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb"}
Oct 31 12:59:59.907446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.907037    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"
Oct 31 12:59:59.907446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.907073    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg" event={"ID":"342f09b7-245d-4235-84da-8625ac04f790","Type":"ContainerDied","Data":"f164a21480ec5aae0af654d96f8ef1f3d5b4ff9c5b82b9b05a29cb7b320dd05e"}
Oct 31 12:59:59.907446 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.907097    5663 scope.go:122] "RemoveContainer" containerID="94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb"
Oct 31 12:59:59.915992 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.915974    5663 scope.go:122] "RemoveContainer" containerID="94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb"
Oct 31 12:59:59.916266 ip-10-0-140-137 kubenswrapper[5663]: E1031 12:59:59.916247    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb\": container with ID starting with 94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb not found: ID does not exist" containerID="94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb"
Oct 31 12:59:59.916320 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.916273    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb"} err="failed to get container status \"94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb\": rpc error: code = NotFound desc = could not find container \"94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb\": container with ID starting with 94c07c9bb3fee7dd93281e9949659eabba97b4cc9a93e9f3d2f2450c054167eb not found: ID does not exist"
Oct 31 12:59:59.926011 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.925989    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"]
Oct 31 12:59:59.933301 ip-10-0-140-137 kubenswrapper[5663]: I1031 12:59:59.933278    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-62734c-openshift-default-7d59dd9b5c-hp7vg"]
Oct 31 13:00:00.149381 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149319    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"]
Oct 31 13:00:00.149775 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149759    5663 cpu_manager.go:414] "containerMap: removing container" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main"
Oct 31 13:00:00.149830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149778    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main"
Oct 31 13:00:00.149830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149789    5663 cpu_manager.go:414] "containerMap: removing container" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy"
Oct 31 13:00:00.149830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149794    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy"
Oct 31 13:00:00.149830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149803    5663 cpu_manager.go:414] "containerMap: removing container" podUID="01469039-f55a-4ea0-ac8d-e5c8b9a3daaa" containerName="payload-processing"
Oct 31 13:00:00.149830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149808    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01469039-f55a-4ea0-ac8d-e5c8b9a3daaa" containerName="payload-processing"
Oct 31 13:00:00.149830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149824    5663 cpu_manager.go:414] "containerMap: removing container" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main"
Oct 31 13:00:00.149830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149828    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149835    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d69e066f-edff-4419-8975-bac11fe1a8f3" containerName="revoke-keys"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149840    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69e066f-edff-4419-8975-bac11fe1a8f3" containerName="revoke-keys"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149846    5663 cpu_manager.go:414] "containerMap: removing container" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149851    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149857    5663 cpu_manager.go:414] "containerMap: removing container" podUID="599d803e-9606-46d0-a5bd-d7b5442e1dc2" containerName="payload-pre-processing"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149862    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599d803e-9606-46d0-a5bd-d7b5442e1dc2" containerName="payload-pre-processing"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149871    5663 cpu_manager.go:414] "containerMap: removing container" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149875    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149947    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69182cf-2356-4273-84be-13030f3405cd" containerName="main"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149957    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="353c883f-02b3-49e1-83bf-e874321bfcca" containerName="istio-proxy"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149962    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="01469039-f55a-4ea0-ac8d-e5c8b9a3daaa" containerName="payload-processing"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149971    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6546f0e-5df8-44d9-9ed1-d7a115a6441a" containerName="main"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149978    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69e066f-edff-4419-8975-bac11fe1a8f3" containerName="revoke-keys"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149985    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149990    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="599d803e-9606-46d0-a5bd-d7b5442e1dc2" containerName="payload-pre-processing"
Oct 31 13:00:00.150037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.149997    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="342f09b7-245d-4235-84da-8625ac04f790" containerName="istio-proxy"
Oct 31 13:00:00.156999 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.156971    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"]
Oct 31 13:00:00.157195 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.157101    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"
Oct 31 13:00:00.167639 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.167538    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"]
Oct 31 13:00:00.167848 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.167825    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"
Oct 31 13:00:00.170941 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.170922    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"]
Oct 31 13:00:00.215495 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.215465    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpgm8\" (UniqueName: \"kubernetes.io/projected/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-kube-api-access-mpgm8\") pod \"maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv\" (UID: \"2eb8d319-2f4f-4f70-b7da-c5fdc217f916\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"
Oct 31 13:00:00.215642 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.215514    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f68e53-cb9c-4a56-88af-92f873997ab3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890860-blm8c\" (UID: \"d3f68e53-cb9c-4a56-88af-92f873997ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"
Oct 31 13:00:00.215686 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.215642    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv\" (UID: \"2eb8d319-2f4f-4f70-b7da-c5fdc217f916\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"
Oct 31 13:00:00.215724 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.215686    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttkn5\" (UniqueName: \"kubernetes.io/projected/d3f68e53-cb9c-4a56-88af-92f873997ab3-kube-api-access-ttkn5\") pod \"maas-api-key-cleanup-29890860-blm8c\" (UID: \"d3f68e53-cb9c-4a56-88af-92f873997ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"
Oct 31 13:00:00.225617 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.225592    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:00:00.316932 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.316896    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpgm8\" (UniqueName: \"kubernetes.io/projected/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-kube-api-access-mpgm8\") pod \"maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv\" (UID: \"2eb8d319-2f4f-4f70-b7da-c5fdc217f916\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"
Oct 31 13:00:00.317070 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.316952    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f68e53-cb9c-4a56-88af-92f873997ab3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890860-blm8c\" (UID: \"d3f68e53-cb9c-4a56-88af-92f873997ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"
Oct 31 13:00:00.317070 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.317007    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv\" (UID: \"2eb8d319-2f4f-4f70-b7da-c5fdc217f916\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"
Oct 31 13:00:00.317070 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.317042    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttkn5\" (UniqueName: \"kubernetes.io/projected/d3f68e53-cb9c-4a56-88af-92f873997ab3-kube-api-access-ttkn5\") pod \"maas-api-key-cleanup-29890860-blm8c\" (UID: \"d3f68e53-cb9c-4a56-88af-92f873997ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"
Oct 31 13:00:00.317671 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.317651    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f68e53-cb9c-4a56-88af-92f873997ab3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890860-blm8c\" (UID: \"d3f68e53-cb9c-4a56-88af-92f873997ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"
Oct 31 13:00:00.317745 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.317691    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv\" (UID: \"2eb8d319-2f4f-4f70-b7da-c5fdc217f916\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"
Oct 31 13:00:00.333934 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.333896    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttkn5\" (UniqueName: \"kubernetes.io/projected/d3f68e53-cb9c-4a56-88af-92f873997ab3-kube-api-access-ttkn5\") pod \"maas-api-key-cleanup-29890860-blm8c\" (UID: \"d3f68e53-cb9c-4a56-88af-92f873997ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"
Oct 31 13:00:00.363411 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.361930    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpgm8\" (UniqueName: \"kubernetes.io/projected/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-kube-api-access-mpgm8\") pod \"maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv\" (UID: \"2eb8d319-2f4f-4f70-b7da-c5fdc217f916\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"
Oct 31 13:00:00.480402 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.480373    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 13:00:00.515320 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.515295    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 13:00:00.518820 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.518794    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab46e85-e4a5-4b1e-a334-1dad88fde563-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab46e85-e4a5-4b1e-a334-1dad88fde563-istiod-ca-cert\") pod \"eab46e85-e4a5-4b1e-a334-1dad88fde563\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") "
Oct 31 13:00:00.518958 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.518833    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-credential-socket\") pod \"eab46e85-e4a5-4b1e-a334-1dad88fde563\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") "
Oct 31 13:00:00.518958 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.518868    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-data\") pod \"eab46e85-e4a5-4b1e-a334-1dad88fde563\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") "
Oct 31 13:00:00.518958 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.518933    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-podinfo\") pod \"eab46e85-e4a5-4b1e-a334-1dad88fde563\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") "
Oct 31 13:00:00.519112 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.518977    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-envoy\") pod \"eab46e85-e4a5-4b1e-a334-1dad88fde563\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") "
Oct 31 13:00:00.519112 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.519018    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-token\" (UniqueName: \"kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-token\") pod \"eab46e85-e4a5-4b1e-a334-1dad88fde563\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") "
Oct 31 13:00:00.519196 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.519111    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-kube-api-access-kjsx8\" (UniqueName: \"kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-kube-api-access-kjsx8\") pod \"eab46e85-e4a5-4b1e-a334-1dad88fde563\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") "
Oct 31 13:00:00.519196 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.519146    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-socket\") pod \"eab46e85-e4a5-4b1e-a334-1dad88fde563\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") "
Oct 31 13:00:00.519607 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.519537    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-socket" pod "eab46e85-e4a5-4b1e-a334-1dad88fde563" (UID: "eab46e85-e4a5-4b1e-a334-1dad88fde563"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:00.519727 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.519608    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-credential-socket" pod "eab46e85-e4a5-4b1e-a334-1dad88fde563" (UID: "eab46e85-e4a5-4b1e-a334-1dad88fde563"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:00.519727 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.519626    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-certs" pod "eab46e85-e4a5-4b1e-a334-1dad88fde563" (UID: "eab46e85-e4a5-4b1e-a334-1dad88fde563"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:00.519727 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.519645    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab46e85-e4a5-4b1e-a334-1dad88fde563-istiod-ca-cert" pod "eab46e85-e4a5-4b1e-a334-1dad88fde563" (UID: "eab46e85-e4a5-4b1e-a334-1dad88fde563"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:00.519727 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.519670    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-data" pod "eab46e85-e4a5-4b1e-a334-1dad88fde563" (UID: "eab46e85-e4a5-4b1e-a334-1dad88fde563"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:00.519923 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.519262    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-certs\") pod \"eab46e85-e4a5-4b1e-a334-1dad88fde563\" (UID: \"eab46e85-e4a5-4b1e-a334-1dad88fde563\") "
Oct 31 13:00:00.520343 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.520319    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.520462 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.520363    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.520462 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.520381    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab46e85-e4a5-4b1e-a334-1dad88fde563-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.520462 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.520395    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.520462 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.520410    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.521873 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.521850    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-token" pod "eab46e85-e4a5-4b1e-a334-1dad88fde563" (UID: "eab46e85-e4a5-4b1e-a334-1dad88fde563"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:00.521980 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.521870    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-podinfo" pod "eab46e85-e4a5-4b1e-a334-1dad88fde563" (UID: "eab46e85-e4a5-4b1e-a334-1dad88fde563"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:00:00.522243 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.522229    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-envoy" pod "eab46e85-e4a5-4b1e-a334-1dad88fde563" (UID: "eab46e85-e4a5-4b1e-a334-1dad88fde563"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:00.522298 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.522274    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-kube-api-access-kjsx8" pod "eab46e85-e4a5-4b1e-a334-1dad88fde563" (UID: "eab46e85-e4a5-4b1e-a334-1dad88fde563"). InnerVolumeSpecName "kube-api-access-kjsx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:00.580291 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.580259    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"
Oct 31 13:00:00.586730 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.586709    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"
Oct 31 13:00:00.621399 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621081    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-kube-api-access-pdlbv\" (UniqueName: \"kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-kube-api-access-pdlbv\") pod \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") "
Oct 31 13:00:00.621399 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621161    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istiod-ca-cert\") pod \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") "
Oct 31 13:00:00.621399 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621198    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-socket\") pod \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") "
Oct 31 13:00:00.621399 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621226    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-podinfo\") pod \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") "
Oct 31 13:00:00.621399 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621267    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-credential-socket\") pod \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") "
Oct 31 13:00:00.621399 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621297    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-envoy\") pod \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") "
Oct 31 13:00:00.621399 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621396    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-token\" (UniqueName: \"kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-token\") pod \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") "
Oct 31 13:00:00.621913 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621446    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-data\") pod \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") "
Oct 31 13:00:00.621913 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621477    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-certs\") pod \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\" (UID: \"c47c3e79-0ee1-4b08-ae9e-87e3507703f5\") "
Oct 31 13:00:00.621913 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621768    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.621913 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621779    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.621913 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621789    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.621913 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.621803    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjsx8\" (UniqueName: \"kubernetes.io/projected/eab46e85-e4a5-4b1e-a334-1dad88fde563-kube-api-access-kjsx8\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.622236 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.622091    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-certs" pod "c47c3e79-0ee1-4b08-ae9e-87e3507703f5" (UID: "c47c3e79-0ee1-4b08-ae9e-87e3507703f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:00.622636 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.622604    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-credential-socket" pod "c47c3e79-0ee1-4b08-ae9e-87e3507703f5" (UID: "c47c3e79-0ee1-4b08-ae9e-87e3507703f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:00.622636 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.622616    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-data" pod "c47c3e79-0ee1-4b08-ae9e-87e3507703f5" (UID: "c47c3e79-0ee1-4b08-ae9e-87e3507703f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:00.622808 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.622721    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-socket" pod "c47c3e79-0ee1-4b08-ae9e-87e3507703f5" (UID: "c47c3e79-0ee1-4b08-ae9e-87e3507703f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:00.623040 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.623016    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istiod-ca-cert" pod "c47c3e79-0ee1-4b08-ae9e-87e3507703f5" (UID: "c47c3e79-0ee1-4b08-ae9e-87e3507703f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:00.624607 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.624577    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-podinfo" pod "c47c3e79-0ee1-4b08-ae9e-87e3507703f5" (UID: "c47c3e79-0ee1-4b08-ae9e-87e3507703f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:00:00.624901 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.624873    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-kube-api-access-pdlbv" pod "c47c3e79-0ee1-4b08-ae9e-87e3507703f5" (UID: "c47c3e79-0ee1-4b08-ae9e-87e3507703f5"). InnerVolumeSpecName "kube-api-access-pdlbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:00.625029 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.624906    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-token" pod "c47c3e79-0ee1-4b08-ae9e-87e3507703f5" (UID: "c47c3e79-0ee1-4b08-ae9e-87e3507703f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:00.625203 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.625182    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-envoy" pod "c47c3e79-0ee1-4b08-ae9e-87e3507703f5" (UID: "c47c3e79-0ee1-4b08-ae9e-87e3507703f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:00.723393 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.723307    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.723393 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.723334    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.723393 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.723343    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.723393 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.723367    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdlbv\" (UniqueName: \"kubernetes.io/projected/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-kube-api-access-pdlbv\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.723393 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.723380    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.723393 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.723389    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.723393 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.723396    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.723714 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.723406    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.723714 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.723414    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c47c3e79-0ee1-4b08-ae9e-87e3507703f5-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:00.869289 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.869248    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"]
Oct 31 13:00:00.871821 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:00:00.871794    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb8d319_2f4f_4f70_b7da_c5fdc217f916.slice/crio-64a9bf25c577f507778683d7a227b8164adda9b628afacb4cca8c00711f3975d WatchSource:0}: Error finding container 64a9bf25c577f507778683d7a227b8164adda9b628afacb4cca8c00711f3975d: Status 404 returned error can't find the container with id 64a9bf25c577f507778683d7a227b8164adda9b628afacb4cca8c00711f3975d
Oct 31 13:00:00.872308 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.872284    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"]
Oct 31 13:00:00.873751 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:00:00.873714    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f68e53_cb9c_4a56_88af_92f873997ab3.slice/crio-4a8c5cdb7280e4818208af48d85529ad9cee0fcfe8081508b47c1efc65037167 WatchSource:0}: Error finding container 4a8c5cdb7280e4818208af48d85529ad9cee0fcfe8081508b47c1efc65037167: Status 404 returned error can't find the container with id 4a8c5cdb7280e4818208af48d85529ad9cee0fcfe8081508b47c1efc65037167
Oct 31 13:00:00.912563 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.912532    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv" event={"ID":"2eb8d319-2f4f-4f70-b7da-c5fdc217f916","Type":"ContainerStarted","Data":"64a9bf25c577f507778683d7a227b8164adda9b628afacb4cca8c00711f3975d"}
Oct 31 13:00:00.913798 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.913772    5663 generic.go:356] "Generic (PLEG): container finished" podID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerID="43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a" exitCode=0
Oct 31 13:00:00.913908 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.913816    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"
Oct 31 13:00:00.913908 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.913846    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" event={"ID":"c47c3e79-0ee1-4b08-ae9e-87e3507703f5","Type":"ContainerDied","Data":"43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a"}
Oct 31 13:00:00.913908 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.913896    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2" event={"ID":"c47c3e79-0ee1-4b08-ae9e-87e3507703f5","Type":"ContainerDied","Data":"a12625931439c1fc310fd524df3bfbfef1e1c56c657d773cb139eda248d69770"}
Oct 31 13:00:00.914023 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.913929    5663 scope.go:122] "RemoveContainer" containerID="43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a"
Oct 31 13:00:00.915167 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.915143    5663 generic.go:356] "Generic (PLEG): container finished" podID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerID="53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1" exitCode=0
Oct 31 13:00:00.915268 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.915201    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"
Oct 31 13:00:00.915268 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.915225    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" event={"ID":"eab46e85-e4a5-4b1e-a334-1dad88fde563","Type":"ContainerDied","Data":"53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1"}
Oct 31 13:00:00.915268 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.915263    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg" event={"ID":"eab46e85-e4a5-4b1e-a334-1dad88fde563","Type":"ContainerDied","Data":"e0de129525cb9696706d98b51da4ea1c9ab09e173d5797745d1af95fc5c2d2e4"}
Oct 31 13:00:00.917634 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.917614    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c" event={"ID":"d3f68e53-cb9c-4a56-88af-92f873997ab3","Type":"ContainerStarted","Data":"4a8c5cdb7280e4818208af48d85529ad9cee0fcfe8081508b47c1efc65037167"}
Oct 31 13:00:00.924154 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.924135    5663 scope.go:122] "RemoveContainer" containerID="43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a"
Oct 31 13:00:00.924488 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:00.924464    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a\": container with ID starting with 43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a not found: ID does not exist" containerID="43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a"
Oct 31 13:00:00.924573 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.924496    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a"} err="failed to get container status \"43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a\": rpc error: code = NotFound desc = could not find container \"43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a\": container with ID starting with 43e7501b59c196a029b658841b1b4ea5cc8a447dc91843379fccc39b200f976a not found: ID does not exist"
Oct 31 13:00:00.924573 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.924512    5663 scope.go:122] "RemoveContainer" containerID="53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1"
Oct 31 13:00:00.934082 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.934067    5663 scope.go:122] "RemoveContainer" containerID="53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1"
Oct 31 13:00:00.934316 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:00.934300    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1\": container with ID starting with 53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1 not found: ID does not exist" containerID="53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1"
Oct 31 13:00:00.934385 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.934324    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1"} err="failed to get container status \"53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1\": rpc error: code = NotFound desc = could not find container \"53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1\": container with ID starting with 53f5825537af8048ea8f7717511c6e3e81f99e9bef50b5c6936b8e6e425d88e1 not found: ID does not exist"
Oct 31 13:00:00.942238 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.942216    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"]
Oct 31 13:00:00.951611 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.951594    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-55qjg"]
Oct 31 13:00:00.962731 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.962711    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"]
Oct 31 13:00:00.967680 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:00.967660    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-lqvc2"]
Oct 31 13:00:01.438960 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:01.438870    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="342f09b7-245d-4235-84da-8625ac04f790" path="/var/lib/kubelet/pods/342f09b7-245d-4235-84da-8625ac04f790/volumes"
Oct 31 13:00:01.439381 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:01.439342    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" path="/var/lib/kubelet/pods/c47c3e79-0ee1-4b08-ae9e-87e3507703f5/volumes"
Oct 31 13:00:01.439785 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:01.439771    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" path="/var/lib/kubelet/pods/eab46e85-e4a5-4b1e-a334-1dad88fde563/volumes"
Oct 31 13:00:01.924839 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:01.924806    5663 generic.go:356] "Generic (PLEG): container finished" podID="d3f68e53-cb9c-4a56-88af-92f873997ab3" containerID="f0fb3f2005d4883a1cc21cb4b43144d780fd304c936e62595ba1fb5fb7675d20" exitCode=60
Oct 31 13:00:01.925314 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:01.924839    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c" event={"ID":"d3f68e53-cb9c-4a56-88af-92f873997ab3","Type":"ContainerDied","Data":"f0fb3f2005d4883a1cc21cb4b43144d780fd304c936e62595ba1fb5fb7675d20"}
Oct 31 13:00:01.925314 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:01.925144    5663 scope.go:122] "RemoveContainer" containerID="f0fb3f2005d4883a1cc21cb4b43144d780fd304c936e62595ba1fb5fb7675d20"
Oct 31 13:00:01.926726 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:01.926655    5663 generic.go:356] "Generic (PLEG): container finished" podID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" containerID="85659dd973d15d7c741fc9fc6f56bca5d658f0e6c7b1ee5352a79d741e86f6c9" exitCode=60
Oct 31 13:00:01.926802 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:01.926739    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv" event={"ID":"2eb8d319-2f4f-4f70-b7da-c5fdc217f916","Type":"ContainerDied","Data":"85659dd973d15d7c741fc9fc6f56bca5d658f0e6c7b1ee5352a79d741e86f6c9"}
Oct 31 13:00:01.926963 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:01.926946    5663 scope.go:122] "RemoveContainer" containerID="85659dd973d15d7c741fc9fc6f56bca5d658f0e6c7b1ee5352a79d741e86f6c9"
Oct 31 13:00:02.937124 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:02.937087    5663 generic.go:356] "Generic (PLEG): container finished" podID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" containerID="2f031de87596e6130150268b6ff42e1adfde01ff625ccba545beb27e1bf4a8fd" exitCode=60
Oct 31 13:00:02.937593 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:02.937172    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv" event={"ID":"2eb8d319-2f4f-4f70-b7da-c5fdc217f916","Type":"ContainerDied","Data":"2f031de87596e6130150268b6ff42e1adfde01ff625ccba545beb27e1bf4a8fd"}
Oct 31 13:00:02.937593 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:02.937228    5663 scope.go:122] "RemoveContainer" containerID="85659dd973d15d7c741fc9fc6f56bca5d658f0e6c7b1ee5352a79d741e86f6c9"
Oct 31 13:00:02.937593 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:02.937510    5663 scope.go:122] "RemoveContainer" containerID="2f031de87596e6130150268b6ff42e1adfde01ff625ccba545beb27e1bf4a8fd"
Oct 31 13:00:02.937819 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:02.937800    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv_odh-ai-gateway-infra(2eb8d319-2f4f-4f70-b7da-c5fdc217f916)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916"
Oct 31 13:00:02.939471 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:02.939178    5663 generic.go:356] "Generic (PLEG): container finished" podID="d3f68e53-cb9c-4a56-88af-92f873997ab3" containerID="4d2228dc0def73efe831aaaf29fabb1c4576176c9c6ce34ffe16f9fb4c77078f" exitCode=60
Oct 31 13:00:02.939471 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:02.939236    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c" event={"ID":"d3f68e53-cb9c-4a56-88af-92f873997ab3","Type":"ContainerDied","Data":"4d2228dc0def73efe831aaaf29fabb1c4576176c9c6ce34ffe16f9fb4c77078f"}
Oct 31 13:00:02.939614 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:02.939542    5663 scope.go:122] "RemoveContainer" containerID="4d2228dc0def73efe831aaaf29fabb1c4576176c9c6ce34ffe16f9fb4c77078f"
Oct 31 13:00:02.939809 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:02.939787    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890860-blm8c_odh-ai-gateway-infra(d3f68e53-cb9c-4a56-88af-92f873997ab3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3"
Oct 31 13:00:02.948648 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:02.948630    5663 scope.go:122] "RemoveContainer" containerID="f0fb3f2005d4883a1cc21cb4b43144d780fd304c936e62595ba1fb5fb7675d20"
Oct 31 13:00:03.946440 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:03.946408    5663 scope.go:122] "RemoveContainer" containerID="4d2228dc0def73efe831aaaf29fabb1c4576176c9c6ce34ffe16f9fb4c77078f"
Oct 31 13:00:03.946949 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:03.946732    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890860-blm8c_odh-ai-gateway-infra(d3f68e53-cb9c-4a56-88af-92f873997ab3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3"
Oct 31 13:00:03.947912 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:03.947894    5663 scope.go:122] "RemoveContainer" containerID="2f031de87596e6130150268b6ff42e1adfde01ff625ccba545beb27e1bf4a8fd"
Oct 31 13:00:03.948142 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:03.948116    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv_odh-ai-gateway-infra(2eb8d319-2f4f-4f70-b7da-c5fdc217f916)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916"
Oct 31 13:00:04.613698 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.613655    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"]
Oct 31 13:00:04.758632 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.758591    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x" podUID="d279e306-b346-462a-a120-9fb77b6dcb0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:00:04.851229 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.851191    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"]
Oct 31 13:00:04.851719 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.851700    5663 cpu_manager.go:414] "containerMap: removing container" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy"
Oct 31 13:00:04.851816 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.851723    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy"
Oct 31 13:00:04.851816 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.851739    5663 cpu_manager.go:414] "containerMap: removing container" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerName="istio-proxy"
Oct 31 13:00:04.851816 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.851749    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerName="istio-proxy"
Oct 31 13:00:04.851974 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.851890    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab46e85-e4a5-4b1e-a334-1dad88fde563" containerName="istio-proxy"
Oct 31 13:00:04.851974 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.851905    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="c47c3e79-0ee1-4b08-ae9e-87e3507703f5" containerName="istio-proxy"
Oct 31 13:00:04.856667 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.856644    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:04.867282 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.867250    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:00:04.873323 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.873291    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"]
Oct 31 13:00:04.989305 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.989262    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvgj6\" (UniqueName: \"kubernetes.io/projected/d3eb289c-05ae-453d-aa54-d06350adee55-kube-api-access-xvgj6\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn\" (UID: \"d3eb289c-05ae-453d-aa54-d06350adee55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:04.989726 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.989420    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn\" (UID: \"d3eb289c-05ae-453d-aa54-d06350adee55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:04.989726 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:04.989594    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn\" (UID: \"d3eb289c-05ae-453d-aa54-d06350adee55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:05.097381 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:05.090652    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn\" (UID: \"d3eb289c-05ae-453d-aa54-d06350adee55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:05.097381 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:05.091707    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvgj6\" (UniqueName: \"kubernetes.io/projected/d3eb289c-05ae-453d-aa54-d06350adee55-kube-api-access-xvgj6\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn\" (UID: \"d3eb289c-05ae-453d-aa54-d06350adee55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:05.097381 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:05.091792    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn\" (UID: \"d3eb289c-05ae-453d-aa54-d06350adee55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:05.097381 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:05.091627    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn\" (UID: \"d3eb289c-05ae-453d-aa54-d06350adee55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:05.097381 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:05.096569    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn\" (UID: \"d3eb289c-05ae-453d-aa54-d06350adee55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:05.107447 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:05.106209    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvgj6\" (UniqueName: \"kubernetes.io/projected/d3eb289c-05ae-453d-aa54-d06350adee55-kube-api-access-xvgj6\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn\" (UID: \"d3eb289c-05ae-453d-aa54-d06350adee55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:05.170312 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:05.170233    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:05.451829 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:05.451802    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"]
Oct 31 13:00:05.454055 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:00:05.454017    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3eb289c_05ae_453d_aa54_d06350adee55.slice/crio-44cc48350b316b77918f6cf1e3b129cf702793f0d8ad1ddb6afb7cabeae042b1 WatchSource:0}: Error finding container 44cc48350b316b77918f6cf1e3b129cf702793f0d8ad1ddb6afb7cabeae042b1: Status 404 returned error can't find the container with id 44cc48350b316b77918f6cf1e3b129cf702793f0d8ad1ddb6afb7cabeae042b1
Oct 31 13:00:05.960411 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:05.960373    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn" event={"ID":"d3eb289c-05ae-453d-aa54-d06350adee55","Type":"ContainerStarted","Data":"4c95289927ac8d5b667e40028c3fa867f8ed34fd37c10337e125a242616a7336"}
Oct 31 13:00:05.960588 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:05.960432    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn" event={"ID":"d3eb289c-05ae-453d-aa54-d06350adee55","Type":"ContainerStarted","Data":"44cc48350b316b77918f6cf1e3b129cf702793f0d8ad1ddb6afb7cabeae042b1"}
Oct 31 13:00:05.982717 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:05.982673    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn" podStartSLOduration=1.9826587 podStartE2EDuration="1.9826587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:05.981607142 +0000 UTC m=+1930.722033358" watchObservedRunningTime="2026-10-31 13:00:05.9826587 +0000 UTC m=+1930.723084915"
Oct 31 13:00:06.979011 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:06.978972    5663 generic.go:356] "Generic (PLEG): container finished" podID="d3eb289c-05ae-453d-aa54-d06350adee55" containerID="4c95289927ac8d5b667e40028c3fa867f8ed34fd37c10337e125a242616a7336" exitCode=0
Oct 31 13:00:06.979539 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:06.979052    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn" event={"ID":"d3eb289c-05ae-453d-aa54-d06350adee55","Type":"ContainerDied","Data":"4c95289927ac8d5b667e40028c3fa867f8ed34fd37c10337e125a242616a7336"}
Oct 31 13:00:07.560174 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.560144    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"
Oct 31 13:00:07.620234 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.620195    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d279e306-b346-462a-a120-9fb77b6dcb0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d279e306-b346-462a-a120-9fb77b6dcb0b-plugins-config-volume\") pod \"d279e306-b346-462a-a120-9fb77b6dcb0b\" (UID: \"d279e306-b346-462a-a120-9fb77b6dcb0b\") "
Oct 31 13:00:07.620425 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.620264    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d279e306-b346-462a-a120-9fb77b6dcb0b-kube-api-access-vnms2\" (UniqueName: \"kubernetes.io/projected/d279e306-b346-462a-a120-9fb77b6dcb0b-kube-api-access-vnms2\") pod \"d279e306-b346-462a-a120-9fb77b6dcb0b\" (UID: \"d279e306-b346-462a-a120-9fb77b6dcb0b\") "
Oct 31 13:00:07.620686 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.620662    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d279e306-b346-462a-a120-9fb77b6dcb0b-plugins-config-volume" pod "d279e306-b346-462a-a120-9fb77b6dcb0b" (UID: "d279e306-b346-462a-a120-9fb77b6dcb0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:07.622514 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.622479    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d279e306-b346-462a-a120-9fb77b6dcb0b-kube-api-access-vnms2" pod "d279e306-b346-462a-a120-9fb77b6dcb0b" (UID: "d279e306-b346-462a-a120-9fb77b6dcb0b"). InnerVolumeSpecName "kube-api-access-vnms2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:07.668245 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.668196    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"]
Oct 31 13:00:07.668753 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.668723    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh" podUID="fb7888a0-116d-4036-98f3-c28c6be8e61e" containerName="istio-proxy" containerID="cri-o://c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2" gracePeriod=30
Oct 31 13:00:07.722288 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.722249    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d279e306-b346-462a-a120-9fb77b6dcb0b-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:07.722288 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.722281    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnms2\" (UniqueName: \"kubernetes.io/projected/d279e306-b346-462a-a120-9fb77b6dcb0b-kube-api-access-vnms2\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:07.986137 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.986097    5663 generic.go:356] "Generic (PLEG): container finished" podID="d279e306-b346-462a-a120-9fb77b6dcb0b" containerID="68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3" exitCode=0
Oct 31 13:00:07.986661 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.986162    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"
Oct 31 13:00:07.986661 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.986189    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x" event={"ID":"d279e306-b346-462a-a120-9fb77b6dcb0b","Type":"ContainerDied","Data":"68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3"}
Oct 31 13:00:07.986661 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.986245    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x" event={"ID":"d279e306-b346-462a-a120-9fb77b6dcb0b","Type":"ContainerDied","Data":"b0261b082d4c2e69dba8cf408a392444f8d17404f4bd4ce0872e2ad7d128dae8"}
Oct 31 13:00:07.986661 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.986275    5663 scope.go:122] "RemoveContainer" containerID="68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3"
Oct 31 13:00:07.995317 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.995293    5663 scope.go:122] "RemoveContainer" containerID="68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3"
Oct 31 13:00:07.995737 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:07.995698    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3\": container with ID starting with 68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3 not found: ID does not exist" containerID="68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3"
Oct 31 13:00:07.995833 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:07.995751    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3"} err="failed to get container status \"68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3\": rpc error: code = NotFound desc = could not find container \"68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3\": container with ID starting with 68d2c53faa6d0d675ded2a894ba786ad0009f1556304e4f4af756bbb4abc09e3 not found: ID does not exist"
Oct 31 13:00:08.027939 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:08.027876    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"]
Oct 31 13:00:08.040782 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:08.040526    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b25d3b-7cf78cbd7-ktp9x"]
Oct 31 13:00:08.151573 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:08.151537    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:08.230449 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:08.230418    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3eb289c-05ae-453d-aa54-d06350adee55-kube-api-access-xvgj6\" (UniqueName: \"kubernetes.io/projected/d3eb289c-05ae-453d-aa54-d06350adee55-kube-api-access-xvgj6\") pod \"d3eb289c-05ae-453d-aa54-d06350adee55\" (UID: \"d3eb289c-05ae-453d-aa54-d06350adee55\") "
Oct 31 13:00:08.230708 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:08.230620    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-xks-ca\") pod \"d3eb289c-05ae-453d-aa54-d06350adee55\" (UID: \"d3eb289c-05ae-453d-aa54-d06350adee55\") "
Oct 31 13:00:08.230708 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:08.230690    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-openshift-ca\") pod \"d3eb289c-05ae-453d-aa54-d06350adee55\" (UID: \"d3eb289c-05ae-453d-aa54-d06350adee55\") "
Oct 31 13:00:08.231114 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:08.231086    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-openshift-ca" pod "d3eb289c-05ae-453d-aa54-d06350adee55" (UID: "d3eb289c-05ae-453d-aa54-d06350adee55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:08.232683 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:08.232647    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-xks-ca" pod "d3eb289c-05ae-453d-aa54-d06350adee55" (UID: "d3eb289c-05ae-453d-aa54-d06350adee55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:08.232795 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:08.232713    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3eb289c-05ae-453d-aa54-d06350adee55-kube-api-access-xvgj6" pod "d3eb289c-05ae-453d-aa54-d06350adee55" (UID: "d3eb289c-05ae-453d-aa54-d06350adee55"). InnerVolumeSpecName "kube-api-access-xvgj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:08.331656 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:08.331622    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvgj6\" (UniqueName: \"kubernetes.io/projected/d3eb289c-05ae-453d-aa54-d06350adee55-kube-api-access-xvgj6\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:08.331656 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:08.331653    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:08.331656 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:08.331664    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3eb289c-05ae-453d-aa54-d06350adee55-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:09.005391 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:09.001148    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn" event={"ID":"d3eb289c-05ae-453d-aa54-d06350adee55","Type":"ContainerDied","Data":"44cc48350b316b77918f6cf1e3b129cf702793f0d8ad1ddb6afb7cabeae042b1"}
Oct 31 13:00:09.005391 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:09.001216    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44cc48350b316b77918f6cf1e3b129cf702793f0d8ad1ddb6afb7cabeae042b1"
Oct 31 13:00:09.005391 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:09.001291    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"
Oct 31 13:00:09.438191 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:09.438155    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d279e306-b346-462a-a120-9fb77b6dcb0b" path="/var/lib/kubelet/pods/d279e306-b346-462a-a120-9fb77b6dcb0b/volumes"
Oct 31 13:00:09.614880 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:09.614833    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl" podUID="8e5921ea-8954-4cf9-be8d-620f35be10e1" containerName="payload-processing" containerID="cri-o://f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9" gracePeriod=25
Oct 31 13:00:09.867303 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:09.867275    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"
Oct 31 13:00:09.958933 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:09.958897    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5921ea-8954-4cf9-be8d-620f35be10e1-kube-api-access-vg2kh\" (UniqueName: \"kubernetes.io/projected/8e5921ea-8954-4cf9-be8d-620f35be10e1-kube-api-access-vg2kh\") pod \"8e5921ea-8954-4cf9-be8d-620f35be10e1\" (UID: \"8e5921ea-8954-4cf9-be8d-620f35be10e1\") "
Oct 31 13:00:09.959115 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:09.958952    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5921ea-8954-4cf9-be8d-620f35be10e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5921ea-8954-4cf9-be8d-620f35be10e1-plugins-config-volume\") pod \"8e5921ea-8954-4cf9-be8d-620f35be10e1\" (UID: \"8e5921ea-8954-4cf9-be8d-620f35be10e1\") "
Oct 31 13:00:09.959410 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:09.959347    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5921ea-8954-4cf9-be8d-620f35be10e1-plugins-config-volume" pod "8e5921ea-8954-4cf9-be8d-620f35be10e1" (UID: "8e5921ea-8954-4cf9-be8d-620f35be10e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:09.961120 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:09.961101    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5921ea-8954-4cf9-be8d-620f35be10e1-kube-api-access-vg2kh" pod "8e5921ea-8954-4cf9-be8d-620f35be10e1" (UID: "8e5921ea-8954-4cf9-be8d-620f35be10e1"). InnerVolumeSpecName "kube-api-access-vg2kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:10.008176 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:10.008080    5663 generic.go:356] "Generic (PLEG): container finished" podID="8e5921ea-8954-4cf9-be8d-620f35be10e1" containerID="f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9" exitCode=0
Oct 31 13:00:10.008176 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:10.008131    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"
Oct 31 13:00:10.008176 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:10.008150    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl" event={"ID":"8e5921ea-8954-4cf9-be8d-620f35be10e1","Type":"ContainerDied","Data":"f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9"}
Oct 31 13:00:10.008715 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:10.008192    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl" event={"ID":"8e5921ea-8954-4cf9-be8d-620f35be10e1","Type":"ContainerDied","Data":"524e889789e7261422dc4f7dab6a368c4f6d227b7ac024ae3cfb014d59bc7f26"}
Oct 31 13:00:10.008715 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:10.008213    5663 scope.go:122] "RemoveContainer" containerID="f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9"
Oct 31 13:00:10.017653 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:10.017636    5663 scope.go:122] "RemoveContainer" containerID="f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9"
Oct 31 13:00:10.017925 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:10.017906    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9\": container with ID starting with f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9 not found: ID does not exist" containerID="f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9"
Oct 31 13:00:10.017994 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:10.017936    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9"} err="failed to get container status \"f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9\": rpc error: code = NotFound desc = could not find container \"f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9\": container with ID starting with f63669b48ab4f1d3475b7b0ad9af787726e29bee79c21029ab263ecbaf6cb8b9 not found: ID does not exist"
Oct 31 13:00:10.030869 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:10.030843    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"]
Oct 31 13:00:10.034632 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:10.034604    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-8faad6-7f4f9bc8f5-gk2nl"]
Oct 31 13:00:10.059854 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:10.059822    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg2kh\" (UniqueName: \"kubernetes.io/projected/8e5921ea-8954-4cf9-be8d-620f35be10e1-kube-api-access-vg2kh\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:10.059854 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:10.059850    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5921ea-8954-4cf9-be8d-620f35be10e1-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:11.442534 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:11.440911    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5921ea-8954-4cf9-be8d-620f35be10e1" path="/var/lib/kubelet/pods/8e5921ea-8954-4cf9-be8d-620f35be10e1/volumes"
Oct 31 13:00:12.929476 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:12.929449    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 13:00:13.027608 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.027533    5663 generic.go:356] "Generic (PLEG): container finished" podID="fb7888a0-116d-4036-98f3-c28c6be8e61e" containerID="c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2" exitCode=0
Oct 31 13:00:13.027608 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.027579    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"
Oct 31 13:00:13.027793 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.027580    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh" event={"ID":"fb7888a0-116d-4036-98f3-c28c6be8e61e","Type":"ContainerDied","Data":"c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2"}
Oct 31 13:00:13.027793 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.027725    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh" event={"ID":"fb7888a0-116d-4036-98f3-c28c6be8e61e","Type":"ContainerDied","Data":"56e74be6a309cc337be7746848c26e5bd2c0f758d19216db7f9403646cd995c1"}
Oct 31 13:00:13.027793 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.027757    5663 scope.go:122] "RemoveContainer" containerID="c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2"
Oct 31 13:00:13.038593 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.038572    5663 scope.go:122] "RemoveContainer" containerID="c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2"
Oct 31 13:00:13.038956 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:13.038926    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2\": container with ID starting with c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2 not found: ID does not exist" containerID="c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2"
Oct 31 13:00:13.039053 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.038970    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2"} err="failed to get container status \"c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2\": rpc error: code = NotFound desc = could not find container \"c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2\": container with ID starting with c32425aa85facdb98033f32c218478ef455346186f076f71888b2dbdedb81bf2 not found: ID does not exist"
Oct 31 13:00:13.097556 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.097524    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-podinfo\") pod \"fb7888a0-116d-4036-98f3-c28c6be8e61e\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") "
Oct 31 13:00:13.097735 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.097584    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-credential-socket\") pod \"fb7888a0-116d-4036-98f3-c28c6be8e61e\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") "
Oct 31 13:00:13.097735 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.097625    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb7888a0-116d-4036-98f3-c28c6be8e61e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7888a0-116d-4036-98f3-c28c6be8e61e-istiod-ca-cert\") pod \"fb7888a0-116d-4036-98f3-c28c6be8e61e\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") "
Oct 31 13:00:13.097735 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.097657    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-certs\") pod \"fb7888a0-116d-4036-98f3-c28c6be8e61e\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") "
Oct 31 13:00:13.097735 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.097683    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-kube-api-access-8p5sd\" (UniqueName: \"kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-kube-api-access-8p5sd\") pod \"fb7888a0-116d-4036-98f3-c28c6be8e61e\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") "
Oct 31 13:00:13.097735 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.097715    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-token\" (UniqueName: \"kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-token\") pod \"fb7888a0-116d-4036-98f3-c28c6be8e61e\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") "
Oct 31 13:00:13.098012 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.097744    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-data\") pod \"fb7888a0-116d-4036-98f3-c28c6be8e61e\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") "
Oct 31 13:00:13.098012 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.097866    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-envoy\") pod \"fb7888a0-116d-4036-98f3-c28c6be8e61e\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") "
Oct 31 13:00:13.098012 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.097926    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-socket\") pod \"fb7888a0-116d-4036-98f3-c28c6be8e61e\" (UID: \"fb7888a0-116d-4036-98f3-c28c6be8e61e\") "
Oct 31 13:00:13.098012 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.097926    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-credential-socket" pod "fb7888a0-116d-4036-98f3-c28c6be8e61e" (UID: "fb7888a0-116d-4036-98f3-c28c6be8e61e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.098239 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.098018    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-certs" pod "fb7888a0-116d-4036-98f3-c28c6be8e61e" (UID: "fb7888a0-116d-4036-98f3-c28c6be8e61e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.098239 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.098056    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7888a0-116d-4036-98f3-c28c6be8e61e-istiod-ca-cert" pod "fb7888a0-116d-4036-98f3-c28c6be8e61e" (UID: "fb7888a0-116d-4036-98f3-c28c6be8e61e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:13.098239 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.098202    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-data" pod "fb7888a0-116d-4036-98f3-c28c6be8e61e" (UID: "fb7888a0-116d-4036-98f3-c28c6be8e61e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.098424 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.098251    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-socket" pod "fb7888a0-116d-4036-98f3-c28c6be8e61e" (UID: "fb7888a0-116d-4036-98f3-c28c6be8e61e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.098488 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.098455    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.098488 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.098470    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.098488 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.098479    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7888a0-116d-4036-98f3-c28c6be8e61e-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.098643 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.098491    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.098643 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.098504    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.099995 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.099963    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-kube-api-access-8p5sd" pod "fb7888a0-116d-4036-98f3-c28c6be8e61e" (UID: "fb7888a0-116d-4036-98f3-c28c6be8e61e"). InnerVolumeSpecName "kube-api-access-8p5sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:13.100108 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.099994    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-envoy" pod "fb7888a0-116d-4036-98f3-c28c6be8e61e" (UID: "fb7888a0-116d-4036-98f3-c28c6be8e61e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:13.100243 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.100225    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-podinfo" pod "fb7888a0-116d-4036-98f3-c28c6be8e61e" (UID: "fb7888a0-116d-4036-98f3-c28c6be8e61e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:00:13.100303 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.100270    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-token" pod "fb7888a0-116d-4036-98f3-c28c6be8e61e" (UID: "fb7888a0-116d-4036-98f3-c28c6be8e61e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:13.199651 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.199616    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.199651 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.199643    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.199651 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.199653    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p5sd\" (UniqueName: \"kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-kube-api-access-8p5sd\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.199888 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.199663    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7888a0-116d-4036-98f3-c28c6be8e61e-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:13.351734 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.351701    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"]
Oct 31 13:00:13.358490 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.358463    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b25d3b-openshift-default-5dfd44db55-f9zjh"]
Oct 31 13:00:13.438301 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:13.438260    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7888a0-116d-4036-98f3-c28c6be8e61e" path="/var/lib/kubelet/pods/fb7888a0-116d-4036-98f3-c28c6be8e61e/volumes"
Oct 31 13:00:14.976142 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:14.976097    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"]
Oct 31 13:00:16.432102 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:16.432072    5663 scope.go:122] "RemoveContainer" containerID="2f031de87596e6130150268b6ff42e1adfde01ff625ccba545beb27e1bf4a8fd"
Oct 31 13:00:16.432526 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:16.432135    5663 scope.go:122] "RemoveContainer" containerID="4d2228dc0def73efe831aaaf29fabb1c4576176c9c6ce34ffe16f9fb4c77078f"
Oct 31 13:00:17.053062 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:17.052967    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c" event={"ID":"d3f68e53-cb9c-4a56-88af-92f873997ab3","Type":"ContainerStarted","Data":"1d54951f2f53edce530eda37dde9da819d6321cfa339cd2f52dd5f92d18cac6e"}
Oct 31 13:00:17.054958 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:17.054929    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv" event={"ID":"2eb8d319-2f4f-4f70-b7da-c5fdc217f916","Type":"ContainerStarted","Data":"09787590b3fab484bea427fb1b73ed5b90595423cf984d65476ff5e04e7306d9"}
Oct 31 13:00:17.085755 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:17.085701    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c" podStartSLOduration=17.085685728 podStartE2EDuration="17.085685728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:17.084654369 +0000 UTC m=+1941.825080620" watchObservedRunningTime="2026-10-31 13:00:17.085685728 +0000 UTC m=+1941.826111944"
Oct 31 13:00:17.106898 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:17.106847    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv" podStartSLOduration=17.106831387 podStartE2EDuration="17.106831387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:17.105965286 +0000 UTC m=+1941.846391502" watchObservedRunningTime="2026-10-31 13:00:17.106831387 +0000 UTC m=+1941.847257603"
Oct 31 13:00:18.085709 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:18.085671    5663 generic.go:356] "Generic (PLEG): container finished" podID="d3f68e53-cb9c-4a56-88af-92f873997ab3" containerID="1d54951f2f53edce530eda37dde9da819d6321cfa339cd2f52dd5f92d18cac6e" exitCode=60
Oct 31 13:00:18.086209 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:18.085742    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c" event={"ID":"d3f68e53-cb9c-4a56-88af-92f873997ab3","Type":"ContainerDied","Data":"1d54951f2f53edce530eda37dde9da819d6321cfa339cd2f52dd5f92d18cac6e"}
Oct 31 13:00:18.086209 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:18.085804    5663 scope.go:122] "RemoveContainer" containerID="4d2228dc0def73efe831aaaf29fabb1c4576176c9c6ce34ffe16f9fb4c77078f"
Oct 31 13:00:18.086209 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:18.086182    5663 scope.go:122] "RemoveContainer" containerID="1d54951f2f53edce530eda37dde9da819d6321cfa339cd2f52dd5f92d18cac6e"
Oct 31 13:00:18.086513 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:18.086492    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890860-blm8c_odh-ai-gateway-infra(d3f68e53-cb9c-4a56-88af-92f873997ab3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3"
Oct 31 13:00:18.088341 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:18.088321    5663 generic.go:356] "Generic (PLEG): container finished" podID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" containerID="09787590b3fab484bea427fb1b73ed5b90595423cf984d65476ff5e04e7306d9" exitCode=60
Oct 31 13:00:18.088473 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:18.088405    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv" event={"ID":"2eb8d319-2f4f-4f70-b7da-c5fdc217f916","Type":"ContainerDied","Data":"09787590b3fab484bea427fb1b73ed5b90595423cf984d65476ff5e04e7306d9"}
Oct 31 13:00:18.088715 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:18.088702    5663 scope.go:122] "RemoveContainer" containerID="09787590b3fab484bea427fb1b73ed5b90595423cf984d65476ff5e04e7306d9"
Oct 31 13:00:18.088922 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:18.088906    5663 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv_odh-ai-gateway-infra(2eb8d319-2f4f-4f70-b7da-c5fdc217f916)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916"
Oct 31 13:00:18.095527 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:18.095503    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"]
Oct 31 13:00:18.098430 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:18.098394    5663 scope.go:122] "RemoveContainer" containerID="2f031de87596e6130150268b6ff42e1adfde01ff625ccba545beb27e1bf4a8fd"
Oct 31 13:00:18.120986 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:18.120961    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"]
Oct 31 13:00:18.142954 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:18.142925    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"]
Oct 31 13:00:18.149376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:18.149326    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2d6201-jqmcn"]
Oct 31 13:00:19.321990 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.321966    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"
Oct 31 13:00:19.343909 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.343884    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"
Oct 31 13:00:19.440063 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.439984    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3eb289c-05ae-453d-aa54-d06350adee55" path="/var/lib/kubelet/pods/d3eb289c-05ae-453d-aa54-d06350adee55/volumes"
Oct 31 13:00:19.512376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.510788    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-maas-api-service-ca\") pod \"2eb8d319-2f4f-4f70-b7da-c5fdc217f916\" (UID: \"2eb8d319-2f4f-4f70-b7da-c5fdc217f916\") "
Oct 31 13:00:19.512376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.510859    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-kube-api-access-mpgm8\" (UniqueName: \"kubernetes.io/projected/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-kube-api-access-mpgm8\") pod \"2eb8d319-2f4f-4f70-b7da-c5fdc217f916\" (UID: \"2eb8d319-2f4f-4f70-b7da-c5fdc217f916\") "
Oct 31 13:00:19.512376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.510924    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f68e53-cb9c-4a56-88af-92f873997ab3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f68e53-cb9c-4a56-88af-92f873997ab3-maas-api-service-ca\") pod \"d3f68e53-cb9c-4a56-88af-92f873997ab3\" (UID: \"d3f68e53-cb9c-4a56-88af-92f873997ab3\") "
Oct 31 13:00:19.512376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.511006    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f68e53-cb9c-4a56-88af-92f873997ab3-kube-api-access-ttkn5\" (UniqueName: \"kubernetes.io/projected/d3f68e53-cb9c-4a56-88af-92f873997ab3-kube-api-access-ttkn5\") pod \"d3f68e53-cb9c-4a56-88af-92f873997ab3\" (UID: \"d3f68e53-cb9c-4a56-88af-92f873997ab3\") "
Oct 31 13:00:19.512766 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.512403    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f68e53-cb9c-4a56-88af-92f873997ab3-maas-api-service-ca" pod "d3f68e53-cb9c-4a56-88af-92f873997ab3" (UID: "d3f68e53-cb9c-4a56-88af-92f873997ab3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:19.512766 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.512678    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-maas-api-service-ca" pod "2eb8d319-2f4f-4f70-b7da-c5fdc217f916" (UID: "2eb8d319-2f4f-4f70-b7da-c5fdc217f916"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:19.523071 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.517517    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f68e53-cb9c-4a56-88af-92f873997ab3-kube-api-access-ttkn5" pod "d3f68e53-cb9c-4a56-88af-92f873997ab3" (UID: "d3f68e53-cb9c-4a56-88af-92f873997ab3"). InnerVolumeSpecName "kube-api-access-ttkn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:19.523071 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.517539    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-kube-api-access-mpgm8" pod "2eb8d319-2f4f-4f70-b7da-c5fdc217f916" (UID: "2eb8d319-2f4f-4f70-b7da-c5fdc217f916"). InnerVolumeSpecName "kube-api-access-mpgm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:19.612788 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.612754    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-maas-api-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:19.612788 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.612783    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpgm8\" (UniqueName: \"kubernetes.io/projected/2eb8d319-2f4f-4f70-b7da-c5fdc217f916-kube-api-access-mpgm8\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:19.612788 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.612793    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f68e53-cb9c-4a56-88af-92f873997ab3-maas-api-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:19.613022 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.612802    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttkn5\" (UniqueName: \"kubernetes.io/projected/d3f68e53-cb9c-4a56-88af-92f873997ab3-kube-api-access-ttkn5\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:19.978669 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:19.978443    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x" podUID="b2ed33ef-f491-4560-b06f-7ae78765e930" containerName="payload-pre-processing" containerID="cri-o://ef1bd38f6fa02b5e95ad6c08e6f62fca88bee25432dfac7d37fb042d86841461" gracePeriod=25
Oct 31 13:00:20.107724 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.107695    5663 generic.go:356] "Generic (PLEG): container finished" podID="b2ed33ef-f491-4560-b06f-7ae78765e930" containerID="ef1bd38f6fa02b5e95ad6c08e6f62fca88bee25432dfac7d37fb042d86841461" exitCode=0
Oct 31 13:00:20.107888 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.107764    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x" event={"ID":"b2ed33ef-f491-4560-b06f-7ae78765e930","Type":"ContainerDied","Data":"ef1bd38f6fa02b5e95ad6c08e6f62fca88bee25432dfac7d37fb042d86841461"}
Oct 31 13:00:20.109208 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.109186    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"
Oct 31 13:00:20.109365 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.109201    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv" event={"ID":"2eb8d319-2f4f-4f70-b7da-c5fdc217f916","Type":"ContainerDied","Data":"64a9bf25c577f507778683d7a227b8164adda9b628afacb4cca8c00711f3975d"}
Oct 31 13:00:20.109365 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.109253    5663 scope.go:122] "RemoveContainer" containerID="09787590b3fab484bea427fb1b73ed5b90595423cf984d65476ff5e04e7306d9"
Oct 31 13:00:20.110625 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.110600    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c" event={"ID":"d3f68e53-cb9c-4a56-88af-92f873997ab3","Type":"ContainerDied","Data":"4a8c5cdb7280e4818208af48d85529ad9cee0fcfe8081508b47c1efc65037167"}
Oct 31 13:00:20.110710 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.110642    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"
Oct 31 13:00:20.124713 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.119655    5663 scope.go:122] "RemoveContainer" containerID="1d54951f2f53edce530eda37dde9da819d6321cfa339cd2f52dd5f92d18cac6e"
Oct 31 13:00:20.153333 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.153293    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"]
Oct 31 13:00:20.176379 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.164868    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890860-blm8c"]
Oct 31 13:00:20.176379 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.175403    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"]
Oct 31 13:00:20.189379 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.188740    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-67d410-29890860-qgbmv"]
Oct 31 13:00:20.297547 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.297520    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"
Oct 31 13:00:20.423762 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.423731    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2ed33ef-f491-4560-b06f-7ae78765e930-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ed33ef-f491-4560-b06f-7ae78765e930-plugins-config-volume\") pod \"b2ed33ef-f491-4560-b06f-7ae78765e930\" (UID: \"b2ed33ef-f491-4560-b06f-7ae78765e930\") "
Oct 31 13:00:20.424149 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.423836    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ed33ef-f491-4560-b06f-7ae78765e930-kube-api-access-jh9zq\" (UniqueName: \"kubernetes.io/projected/b2ed33ef-f491-4560-b06f-7ae78765e930-kube-api-access-jh9zq\") pod \"b2ed33ef-f491-4560-b06f-7ae78765e930\" (UID: \"b2ed33ef-f491-4560-b06f-7ae78765e930\") "
Oct 31 13:00:20.424149 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.424086    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ed33ef-f491-4560-b06f-7ae78765e930-plugins-config-volume" pod "b2ed33ef-f491-4560-b06f-7ae78765e930" (UID: "b2ed33ef-f491-4560-b06f-7ae78765e930"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:20.426011 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.425983    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ed33ef-f491-4560-b06f-7ae78765e930-kube-api-access-jh9zq" pod "b2ed33ef-f491-4560-b06f-7ae78765e930" (UID: "b2ed33ef-f491-4560-b06f-7ae78765e930"). InnerVolumeSpecName "kube-api-access-jh9zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:20.525594 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.525559    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ed33ef-f491-4560-b06f-7ae78765e930-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:20.525594 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:20.525587    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh9zq\" (UniqueName: \"kubernetes.io/projected/b2ed33ef-f491-4560-b06f-7ae78765e930-kube-api-access-jh9zq\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:21.118826 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:21.118786    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x" event={"ID":"b2ed33ef-f491-4560-b06f-7ae78765e930","Type":"ContainerDied","Data":"82c1ed97342a683970920d94cc01427f2c5b774a349d7de602bf69162cb4d369"}
Oct 31 13:00:21.119022 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:21.118857    5663 scope.go:122] "RemoveContainer" containerID="ef1bd38f6fa02b5e95ad6c08e6f62fca88bee25432dfac7d37fb042d86841461"
Oct 31 13:00:21.119022 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:21.118801    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"
Oct 31 13:00:21.142125 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:21.142095    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"]
Oct 31 13:00:21.147856 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:21.147834    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2d6201-bdfd4df95-m7m9x"]
Oct 31 13:00:21.437467 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:21.437396    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" path="/var/lib/kubelet/pods/2eb8d319-2f4f-4f70-b7da-c5fdc217f916/volumes"
Oct 31 13:00:21.437822 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:21.437771    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ed33ef-f491-4560-b06f-7ae78765e930" path="/var/lib/kubelet/pods/b2ed33ef-f491-4560-b06f-7ae78765e930/volumes"
Oct 31 13:00:21.438097 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:21.438083    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3" path="/var/lib/kubelet/pods/d3f68e53-cb9c-4a56-88af-92f873997ab3/volumes"
Oct 31 13:00:22.476397 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:22.476338    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:00:25.154490 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:25.154456    5663 generic.go:356] "Generic (PLEG): container finished" podID="e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" containerID="0b3beb4ccacf597afcdb510af4d624269171c71e59558473a1285a91a3d29011" exitCode=0
Oct 31 13:00:25.155085 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:25.154532    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts" event={"ID":"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761","Type":"ContainerDied","Data":"0b3beb4ccacf597afcdb510af4d624269171c71e59558473a1285a91a3d29011"}
Oct 31 13:00:25.155085 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:25.155031    5663 scope.go:122] "RemoveContainer" containerID="0b3beb4ccacf597afcdb510af4d624269171c71e59558473a1285a91a3d29011"
Oct 31 13:00:25.156281 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:25.156253    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:00:25.156389 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:25.156324    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:00:25.156389 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:25.156369    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:00:26.033019 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:26.032978    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"]
Oct 31 13:00:26.163904 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:26.163869    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts" event={"ID":"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761","Type":"ContainerStarted","Data":"a3ad55844cdcc2cbe00180eb314234e8705a54671645ea9ef1e15a1af158ff29"}
Oct 31 13:00:26.164686 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:26.164080    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts" podUID="e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" containerName="istio-proxy" containerID="cri-o://a3ad55844cdcc2cbe00180eb314234e8705a54671645ea9ef1e15a1af158ff29" gracePeriod=30
Oct 31 13:00:29.172067 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172025    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"]
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172431    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d279e306-b346-462a-a120-9fb77b6dcb0b" containerName="payload-pre-processing"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172444    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d279e306-b346-462a-a120-9fb77b6dcb0b" containerName="payload-pre-processing"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172455    5663 cpu_manager.go:414] "containerMap: removing container" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" containerName="cleanup"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172460    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" containerName="cleanup"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172468    5663 cpu_manager.go:414] "containerMap: removing container" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" containerName="cleanup"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172473    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" containerName="cleanup"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172482    5663 cpu_manager.go:414] "containerMap: removing container" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" containerName="cleanup"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172486    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" containerName="cleanup"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172499    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3" containerName="cleanup"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172503    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3" containerName="cleanup"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172509    5663 cpu_manager.go:414] "containerMap: removing container" podUID="b2ed33ef-f491-4560-b06f-7ae78765e930" containerName="payload-pre-processing"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172513    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ed33ef-f491-4560-b06f-7ae78765e930" containerName="payload-pre-processing"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172525    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3" containerName="cleanup"
Oct 31 13:00:29.172521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172530    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3" containerName="cleanup"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172536    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3" containerName="cleanup"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172540    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3" containerName="cleanup"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172555    5663 cpu_manager.go:414] "containerMap: removing container" podUID="fb7888a0-116d-4036-98f3-c28c6be8e61e" containerName="istio-proxy"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172559    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7888a0-116d-4036-98f3-c28c6be8e61e" containerName="istio-proxy"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172565    5663 cpu_manager.go:414] "containerMap: removing container" podUID="d3eb289c-05ae-453d-aa54-d06350adee55" containerName="revoke-keys"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172570    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3eb289c-05ae-453d-aa54-d06350adee55" containerName="revoke-keys"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172575    5663 cpu_manager.go:414] "containerMap: removing container" podUID="8e5921ea-8954-4cf9-be8d-620f35be10e1" containerName="payload-processing"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172579    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5921ea-8954-4cf9-be8d-620f35be10e1" containerName="payload-processing"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172626    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3" containerName="cleanup"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172632    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3" containerName="cleanup"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172642    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7888a0-116d-4036-98f3-c28c6be8e61e" containerName="istio-proxy"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172648    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3eb289c-05ae-453d-aa54-d06350adee55" containerName="revoke-keys"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172653    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5921ea-8954-4cf9-be8d-620f35be10e1" containerName="payload-processing"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172659    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d279e306-b346-462a-a120-9fb77b6dcb0b" containerName="payload-pre-processing"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172665    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" containerName="cleanup"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172671    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" containerName="cleanup"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172676    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb8d319-2f4f-4f70-b7da-c5fdc217f916" containerName="cleanup"
Oct 31 13:00:29.173605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.172682    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ed33ef-f491-4560-b06f-7ae78765e930" containerName="payload-pre-processing"
Oct 31 13:00:29.176955 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.176935    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:29.188448 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.188421    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"]
Oct 31 13:00:29.216711 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.216682    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422a0516-aa07-47c6-9acd-e56123690393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603e53c1-b9rpn\" (UID: \"422a0516-aa07-47c6-9acd-e56123690393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:29.216711 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.216715    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226g8\" (UniqueName: \"kubernetes.io/projected/422a0516-aa07-47c6-9acd-e56123690393-kube-api-access-226g8\") pod \"maas-api-revoke-sub-603e53c1-b9rpn\" (UID: \"422a0516-aa07-47c6-9acd-e56123690393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:29.216883 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.216745    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422a0516-aa07-47c6-9acd-e56123690393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603e53c1-b9rpn\" (UID: \"422a0516-aa07-47c6-9acd-e56123690393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:29.274970 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.274942    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:00:29.275139 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.275050    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:00:29.318068 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.318039    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422a0516-aa07-47c6-9acd-e56123690393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603e53c1-b9rpn\" (UID: \"422a0516-aa07-47c6-9acd-e56123690393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:29.318253 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.318077    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226g8\" (UniqueName: \"kubernetes.io/projected/422a0516-aa07-47c6-9acd-e56123690393-kube-api-access-226g8\") pod \"maas-api-revoke-sub-603e53c1-b9rpn\" (UID: \"422a0516-aa07-47c6-9acd-e56123690393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:29.318253 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.318120    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422a0516-aa07-47c6-9acd-e56123690393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603e53c1-b9rpn\" (UID: \"422a0516-aa07-47c6-9acd-e56123690393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:29.318771 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.318750    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422a0516-aa07-47c6-9acd-e56123690393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603e53c1-b9rpn\" (UID: \"422a0516-aa07-47c6-9acd-e56123690393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:29.320337 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.320315    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422a0516-aa07-47c6-9acd-e56123690393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603e53c1-b9rpn\" (UID: \"422a0516-aa07-47c6-9acd-e56123690393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:29.331384 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.331336    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-226g8\" (UniqueName: \"kubernetes.io/projected/422a0516-aa07-47c6-9acd-e56123690393-kube-api-access-226g8\") pod \"maas-api-revoke-sub-603e53c1-b9rpn\" (UID: \"422a0516-aa07-47c6-9acd-e56123690393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:29.494644 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.494550    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:29.775928 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:29.775838    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"]
Oct 31 13:00:29.779187 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:00:29.779155    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422a0516_aa07_47c6_9acd_e56123690393.slice/crio-071a6717e471ea00b73aac9eadd1de1b2875305a6a5cc35039aa8e92c73168c0 WatchSource:0}: Error finding container 071a6717e471ea00b73aac9eadd1de1b2875305a6a5cc35039aa8e92c73168c0: Status 404 returned error can't find the container with id 071a6717e471ea00b73aac9eadd1de1b2875305a6a5cc35039aa8e92c73168c0
Oct 31 13:00:30.187110 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:30.187045    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn" event={"ID":"422a0516-aa07-47c6-9acd-e56123690393","Type":"ContainerStarted","Data":"e04676dac356f53bb192ffc06a34a67522e830c81c00b8776471dffeadcb9d7e"}
Oct 31 13:00:30.187110 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:30.187097    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn" event={"ID":"422a0516-aa07-47c6-9acd-e56123690393","Type":"ContainerStarted","Data":"071a6717e471ea00b73aac9eadd1de1b2875305a6a5cc35039aa8e92c73168c0"}
Oct 31 13:00:30.208002 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:30.207952    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn" podStartSLOduration=1.20793822 podStartE2EDuration="1.20793822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:30.206520885 +0000 UTC m=+1954.946947102" watchObservedRunningTime="2026-10-31 13:00:30.20793822 +0000 UTC m=+1954.948364436"
Oct 31 13:00:31.194377 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.194333    5663 generic.go:356] "Generic (PLEG): container finished" podID="422a0516-aa07-47c6-9acd-e56123690393" containerID="e04676dac356f53bb192ffc06a34a67522e830c81c00b8776471dffeadcb9d7e" exitCode=0
Oct 31 13:00:31.194749 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.194396    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn" event={"ID":"422a0516-aa07-47c6-9acd-e56123690393","Type":"ContainerDied","Data":"e04676dac356f53bb192ffc06a34a67522e830c81c00b8776471dffeadcb9d7e"}
Oct 31 13:00:31.196497 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.196473    5663 generic.go:356] "Generic (PLEG): container finished" podID="e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" containerID="a3ad55844cdcc2cbe00180eb314234e8705a54671645ea9ef1e15a1af158ff29" exitCode=0
Oct 31 13:00:31.196616 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.196518    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts" event={"ID":"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761","Type":"ContainerDied","Data":"a3ad55844cdcc2cbe00180eb314234e8705a54671645ea9ef1e15a1af158ff29"}
Oct 31 13:00:31.196616 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.196562    5663 scope.go:122] "RemoveContainer" containerID="0b3beb4ccacf597afcdb510af4d624269171c71e59558473a1285a91a3d29011"
Oct 31 13:00:31.815717 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.815694    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 13:00:31.843115 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843083    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-data\") pod \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") "
Oct 31 13:00:31.843279 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843130    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-socket\") pod \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") "
Oct 31 13:00:31.843279 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843171    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-credential-socket\") pod \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") "
Oct 31 13:00:31.843279 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843236    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-token\" (UniqueName: \"kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-token\") pod \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") "
Oct 31 13:00:31.843492 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843283    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-certs\") pod \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") "
Oct 31 13:00:31.843492 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843313    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istiod-ca-cert\") pod \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") "
Oct 31 13:00:31.843492 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843379    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-data" pod "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" (UID: "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:31.843492 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843388    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-podinfo\") pod \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") "
Oct 31 13:00:31.843492 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843446    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-kube-api-access-qc77r\" (UniqueName: \"kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-kube-api-access-qc77r\") pod \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") "
Oct 31 13:00:31.843762 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843492    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-socket" pod "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" (UID: "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:31.843762 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843517    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-envoy\") pod \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\" (UID: \"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761\") "
Oct 31 13:00:31.843762 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843654    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-credential-socket" pod "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" (UID: "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:31.843762 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843680    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istiod-ca-cert" pod "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" (UID: "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:31.844051 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843846    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:31.844051 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843862    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:31.844051 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843875    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:31.844051 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.843888    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:31.844628 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.844557    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-certs" pod "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" (UID: "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:31.846012 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.845971    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-envoy" pod "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" (UID: "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:31.846287 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.846260    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-kube-api-access-qc77r" pod "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" (UID: "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761"). InnerVolumeSpecName "kube-api-access-qc77r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:31.850667 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.850595    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-token" pod "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" (UID: "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:31.852736 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.852689    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-podinfo" pod "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" (UID: "e5bdfa24-6e4e-4d2c-a4ac-2c515e104761"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:00:31.944899 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.944813    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:31.944899 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.944845    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:31.944899 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.944856    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:31.944899 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.944866    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc77r\" (UniqueName: \"kubernetes.io/projected/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-kube-api-access-qc77r\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:31.944899 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:31.944876    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:32.205936 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.205841    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"
Oct 31 13:00:32.205936 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.205856    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts" event={"ID":"e5bdfa24-6e4e-4d2c-a4ac-2c515e104761","Type":"ContainerDied","Data":"2c58c9310e0df5505771fc42da8cb5b1bc1df4741ec515784964cdd3c255092b"}
Oct 31 13:00:32.205936 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.205930    5663 scope.go:122] "RemoveContainer" containerID="a3ad55844cdcc2cbe00180eb314234e8705a54671645ea9ef1e15a1af158ff29"
Oct 31 13:00:32.232553 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.232524    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"]
Oct 31 13:00:32.238513 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.238487    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2d6201-openshift-default-6bd5d895b7-kxnts"]
Oct 31 13:00:32.340330 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.340304    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:32.449862 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.449828    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422a0516-aa07-47c6-9acd-e56123690393-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422a0516-aa07-47c6-9acd-e56123690393-maas-api-openshift-ca\") pod \"422a0516-aa07-47c6-9acd-e56123690393\" (UID: \"422a0516-aa07-47c6-9acd-e56123690393\") "
Oct 31 13:00:32.450039 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.449873    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422a0516-aa07-47c6-9acd-e56123690393-kube-api-access-226g8\" (UniqueName: \"kubernetes.io/projected/422a0516-aa07-47c6-9acd-e56123690393-kube-api-access-226g8\") pod \"422a0516-aa07-47c6-9acd-e56123690393\" (UID: \"422a0516-aa07-47c6-9acd-e56123690393\") "
Oct 31 13:00:32.450039 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.449919    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422a0516-aa07-47c6-9acd-e56123690393-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422a0516-aa07-47c6-9acd-e56123690393-maas-api-xks-ca\") pod \"422a0516-aa07-47c6-9acd-e56123690393\" (UID: \"422a0516-aa07-47c6-9acd-e56123690393\") "
Oct 31 13:00:32.450192 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.450168    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422a0516-aa07-47c6-9acd-e56123690393-maas-api-openshift-ca" pod "422a0516-aa07-47c6-9acd-e56123690393" (UID: "422a0516-aa07-47c6-9acd-e56123690393"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:32.451976 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.451957    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422a0516-aa07-47c6-9acd-e56123690393-maas-api-xks-ca" pod "422a0516-aa07-47c6-9acd-e56123690393" (UID: "422a0516-aa07-47c6-9acd-e56123690393"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:32.452063 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.452038    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422a0516-aa07-47c6-9acd-e56123690393-kube-api-access-226g8" pod "422a0516-aa07-47c6-9acd-e56123690393" (UID: "422a0516-aa07-47c6-9acd-e56123690393"). InnerVolumeSpecName "kube-api-access-226g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:32.556802 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.556763    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/422a0516-aa07-47c6-9acd-e56123690393-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:32.556802 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.556802    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-226g8\" (UniqueName: \"kubernetes.io/projected/422a0516-aa07-47c6-9acd-e56123690393-kube-api-access-226g8\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:32.557144 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:32.556821    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/422a0516-aa07-47c6-9acd-e56123690393-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:33.219774 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:33.219738    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"
Oct 31 13:00:33.219774 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:33.219754    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn" event={"ID":"422a0516-aa07-47c6-9acd-e56123690393","Type":"ContainerDied","Data":"071a6717e471ea00b73aac9eadd1de1b2875305a6a5cc35039aa8e92c73168c0"}
Oct 31 13:00:33.220307 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:33.219846    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="071a6717e471ea00b73aac9eadd1de1b2875305a6a5cc35039aa8e92c73168c0"
Oct 31 13:00:33.437300 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:33.437263    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" path="/var/lib/kubelet/pods/e5bdfa24-6e4e-4d2c-a4ac-2c515e104761/volumes"
Oct 31 13:00:41.505432 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:41.505392    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"]
Oct 31 13:00:41.529166 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:41.529131    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"]
Oct 31 13:00:47.363678 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.363641    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"]
Oct 31 13:00:47.364227 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.364207    5663 cpu_manager.go:414] "containerMap: removing container" podUID="e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" containerName="istio-proxy"
Oct 31 13:00:47.364314 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.364230    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" containerName="istio-proxy"
Oct 31 13:00:47.364314 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.364273    5663 cpu_manager.go:414] "containerMap: removing container" podUID="422a0516-aa07-47c6-9acd-e56123690393" containerName="revoke-keys"
Oct 31 13:00:47.364314 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.364281    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422a0516-aa07-47c6-9acd-e56123690393" containerName="revoke-keys"
Oct 31 13:00:47.364501 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.364382    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" containerName="istio-proxy"
Oct 31 13:00:47.364501 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.364401    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" containerName="istio-proxy"
Oct 31 13:00:47.364501 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.364416    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f68e53-cb9c-4a56-88af-92f873997ab3" containerName="cleanup"
Oct 31 13:00:47.364501 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.364427    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="422a0516-aa07-47c6-9acd-e56123690393" containerName="revoke-keys"
Oct 31 13:00:47.367502 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.367481    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:47.383227 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.383198    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"]
Oct 31 13:00:47.398554 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.398531    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:00:47.398789 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.398770    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:00:47.514681 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.514642    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngjk\" (UniqueName: \"kubernetes.io/projected/a79edd95-23ec-420c-9fae-73f2623175fd-kube-api-access-jngjk\") pod \"maas-api-revoke-sub-98638026-n4f27\" (UID: \"a79edd95-23ec-420c-9fae-73f2623175fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:47.514870 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.514756    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98638026-n4f27\" (UID: \"a79edd95-23ec-420c-9fae-73f2623175fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:47.514946 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.514889    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98638026-n4f27\" (UID: \"a79edd95-23ec-420c-9fae-73f2623175fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:47.620082 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.619989    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98638026-n4f27\" (UID: \"a79edd95-23ec-420c-9fae-73f2623175fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:47.621218 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.621188    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98638026-n4f27\" (UID: \"a79edd95-23ec-420c-9fae-73f2623175fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:47.621410 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.621249    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jngjk\" (UniqueName: \"kubernetes.io/projected/a79edd95-23ec-420c-9fae-73f2623175fd-kube-api-access-jngjk\") pod \"maas-api-revoke-sub-98638026-n4f27\" (UID: \"a79edd95-23ec-420c-9fae-73f2623175fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:47.621714 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.621047    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98638026-n4f27\" (UID: \"a79edd95-23ec-420c-9fae-73f2623175fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:47.626701 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.626661    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98638026-n4f27\" (UID: \"a79edd95-23ec-420c-9fae-73f2623175fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:47.636883 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.636845    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngjk\" (UniqueName: \"kubernetes.io/projected/a79edd95-23ec-420c-9fae-73f2623175fd-kube-api-access-jngjk\") pod \"maas-api-revoke-sub-98638026-n4f27\" (UID: \"a79edd95-23ec-420c-9fae-73f2623175fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:47.685544 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.685506    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:47.971686 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:47.971596    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"]
Oct 31 13:00:47.977247 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:00:47.977219    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79edd95_23ec_420c_9fae_73f2623175fd.slice/crio-9b7c819ab871abd976dc004d80f3a17c9d7c66987812ece017a0358949b63e3b WatchSource:0}: Error finding container 9b7c819ab871abd976dc004d80f3a17c9d7c66987812ece017a0358949b63e3b: Status 404 returned error can't find the container with id 9b7c819ab871abd976dc004d80f3a17c9d7c66987812ece017a0358949b63e3b
Oct 31 13:00:48.301074 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:48.301037    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27" event={"ID":"a79edd95-23ec-420c-9fae-73f2623175fd","Type":"ContainerStarted","Data":"e4ffcfadce8dddd8ac106dbdefacf00190d72c58aa25ec4c3ed665e4ceefd92d"}
Oct 31 13:00:48.301218 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:48.301100    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27" event={"ID":"a79edd95-23ec-420c-9fae-73f2623175fd","Type":"ContainerStarted","Data":"9b7c819ab871abd976dc004d80f3a17c9d7c66987812ece017a0358949b63e3b"}
Oct 31 13:00:48.323257 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:48.323198    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27" podStartSLOduration=1.323185019 podStartE2EDuration="1.323185019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:48.323036835 +0000 UTC m=+1973.063463052" watchObservedRunningTime="2026-10-31 13:00:48.323185019 +0000 UTC m=+1973.063611234"
Oct 31 13:00:49.307777 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:49.307747    5663 generic.go:356] "Generic (PLEG): container finished" podID="a79edd95-23ec-420c-9fae-73f2623175fd" containerID="e4ffcfadce8dddd8ac106dbdefacf00190d72c58aa25ec4c3ed665e4ceefd92d" exitCode=0
Oct 31 13:00:49.308140 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:49.307830    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27" event={"ID":"a79edd95-23ec-420c-9fae-73f2623175fd","Type":"ContainerDied","Data":"e4ffcfadce8dddd8ac106dbdefacf00190d72c58aa25ec4c3ed665e4ceefd92d"}
Oct 31 13:00:50.448124 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:50.448096    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:50.551284 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:50.551253    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79edd95-23ec-420c-9fae-73f2623175fd-kube-api-access-jngjk\" (UniqueName: \"kubernetes.io/projected/a79edd95-23ec-420c-9fae-73f2623175fd-kube-api-access-jngjk\") pod \"a79edd95-23ec-420c-9fae-73f2623175fd\" (UID: \"a79edd95-23ec-420c-9fae-73f2623175fd\") "
Oct 31 13:00:50.551516 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:50.551297    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-openshift-ca\") pod \"a79edd95-23ec-420c-9fae-73f2623175fd\" (UID: \"a79edd95-23ec-420c-9fae-73f2623175fd\") "
Oct 31 13:00:50.551516 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:50.551321    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-xks-ca\") pod \"a79edd95-23ec-420c-9fae-73f2623175fd\" (UID: \"a79edd95-23ec-420c-9fae-73f2623175fd\") "
Oct 31 13:00:50.551725 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:50.551701    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-openshift-ca" pod "a79edd95-23ec-420c-9fae-73f2623175fd" (UID: "a79edd95-23ec-420c-9fae-73f2623175fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:50.551912 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:50.551897    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:50.553448 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:50.553425    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-xks-ca" pod "a79edd95-23ec-420c-9fae-73f2623175fd" (UID: "a79edd95-23ec-420c-9fae-73f2623175fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:50.553548 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:50.553485    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79edd95-23ec-420c-9fae-73f2623175fd-kube-api-access-jngjk" pod "a79edd95-23ec-420c-9fae-73f2623175fd" (UID: "a79edd95-23ec-420c-9fae-73f2623175fd"). InnerVolumeSpecName "kube-api-access-jngjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:50.659518 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:50.654029    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a79edd95-23ec-420c-9fae-73f2623175fd-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:50.659518 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:50.654068    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jngjk\" (UniqueName: \"kubernetes.io/projected/a79edd95-23ec-420c-9fae-73f2623175fd-kube-api-access-jngjk\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:51.318878 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:51.318840    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"
Oct 31 13:00:51.319050 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:51.318871    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27" event={"ID":"a79edd95-23ec-420c-9fae-73f2623175fd","Type":"ContainerDied","Data":"9b7c819ab871abd976dc004d80f3a17c9d7c66987812ece017a0358949b63e3b"}
Oct 31 13:00:51.319050 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:51.318950    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b7c819ab871abd976dc004d80f3a17c9d7c66987812ece017a0358949b63e3b"
Oct 31 13:00:52.010729 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:52.010687    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"]
Oct 31 13:00:52.012624 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:52.012563    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" containerName="istio-proxy" containerID="cri-o://8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c" gracePeriod=30
Oct 31 13:00:52.475811 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:52.475775    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:00:56.525767 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.525706    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" containerID="cri-o://a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918" gracePeriod=45
Oct 31 13:00:56.549541 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.549506    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main" containerID="cri-o://380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761" gracePeriod=45
Oct 31 13:00:56.793169 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.793142    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 13:00:56.820430 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.820399    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 13:00:56.917704 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.917665    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-tmp-dir\") pod \"f0b4371b-f550-4a21-8c82-f2d84f836b67\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") "
Oct 31 13:00:56.917905 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.917726    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-model-cache\") pod \"f0b4371b-f550-4a21-8c82-f2d84f836b67\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") "
Oct 31 13:00:56.917905 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.917808    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tmp-dir\") pod \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") "
Oct 31 13:00:56.917905 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.917838    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-dshm\") pod \"f0b4371b-f550-4a21-8c82-f2d84f836b67\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") "
Oct 31 13:00:56.917905 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.917878    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-home\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-home\") pod \"f0b4371b-f550-4a21-8c82-f2d84f836b67\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") "
Oct 31 13:00:56.918139 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.917920    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0b4371b-f550-4a21-8c82-f2d84f836b67-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b4371b-f550-4a21-8c82-f2d84f836b67-tls-certs\") pod \"f0b4371b-f550-4a21-8c82-f2d84f836b67\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") "
Oct 31 13:00:56.918139 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.917957    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-model-cache\") pod \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") "
Oct 31 13:00:56.918139 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.917996    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-home\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-home\") pod \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") "
Oct 31 13:00:56.918139 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.917998    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-tmp-dir" pod "f0b4371b-f550-4a21-8c82-f2d84f836b67" (UID: "f0b4371b-f550-4a21-8c82-f2d84f836b67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:56.918139 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918037    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-model-cache" pod "f0b4371b-f550-4a21-8c82-f2d84f836b67" (UID: "f0b4371b-f550-4a21-8c82-f2d84f836b67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:56.918139 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918070    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-dshm\") pod \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") "
Oct 31 13:00:56.918139 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918088    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tmp-dir" pod "5ab6a659-425e-493a-b02d-9a1a23c06aa1" (UID: "5ab6a659-425e-493a-b02d-9a1a23c06aa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:56.918570 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918160    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab6a659-425e-493a-b02d-9a1a23c06aa1-kube-api-access-zwswq\" (UniqueName: \"kubernetes.io/projected/5ab6a659-425e-493a-b02d-9a1a23c06aa1-kube-api-access-zwswq\") pod \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") "
Oct 31 13:00:56.918570 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918227    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b4371b-f550-4a21-8c82-f2d84f836b67-kube-api-access-745w8\" (UniqueName: \"kubernetes.io/projected/f0b4371b-f550-4a21-8c82-f2d84f836b67-kube-api-access-745w8\") pod \"f0b4371b-f550-4a21-8c82-f2d84f836b67\" (UID: \"f0b4371b-f550-4a21-8c82-f2d84f836b67\") "
Oct 31 13:00:56.918570 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918293    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tls-certs\") pod \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\" (UID: \"5ab6a659-425e-493a-b02d-9a1a23c06aa1\") "
Oct 31 13:00:56.918570 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918307    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-home" pod "5ab6a659-425e-493a-b02d-9a1a23c06aa1" (UID: "5ab6a659-425e-493a-b02d-9a1a23c06aa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:56.918788 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918605    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-home" pod "f0b4371b-f550-4a21-8c82-f2d84f836b67" (UID: "f0b4371b-f550-4a21-8c82-f2d84f836b67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:56.918788 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918725    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:56.918788 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918745    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:56.918788 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918761    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:56.918788 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918775    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:56.918788 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918786    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:56.919169 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.918861    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-model-cache" pod "5ab6a659-425e-493a-b02d-9a1a23c06aa1" (UID: "5ab6a659-425e-493a-b02d-9a1a23c06aa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:56.920480 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.920423    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b4371b-f550-4a21-8c82-f2d84f836b67-tls-certs" pod "f0b4371b-f550-4a21-8c82-f2d84f836b67" (UID: "f0b4371b-f550-4a21-8c82-f2d84f836b67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:56.920606 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.920482    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-dshm" pod "f0b4371b-f550-4a21-8c82-f2d84f836b67" (UID: "f0b4371b-f550-4a21-8c82-f2d84f836b67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:56.920883 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.920850    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b4371b-f550-4a21-8c82-f2d84f836b67-kube-api-access-745w8" pod "f0b4371b-f550-4a21-8c82-f2d84f836b67" (UID: "f0b4371b-f550-4a21-8c82-f2d84f836b67"). InnerVolumeSpecName "kube-api-access-745w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:56.920945 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.920930    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-dshm" pod "5ab6a659-425e-493a-b02d-9a1a23c06aa1" (UID: "5ab6a659-425e-493a-b02d-9a1a23c06aa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:56.920991 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.920934    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tls-certs" pod "5ab6a659-425e-493a-b02d-9a1a23c06aa1" (UID: "5ab6a659-425e-493a-b02d-9a1a23c06aa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:56.921084 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:56.921057    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab6a659-425e-493a-b02d-9a1a23c06aa1-kube-api-access-zwswq" pod "5ab6a659-425e-493a-b02d-9a1a23c06aa1" (UID: "5ab6a659-425e-493a-b02d-9a1a23c06aa1"). InnerVolumeSpecName "kube-api-access-zwswq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:57.019593 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.019550    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b4371b-f550-4a21-8c82-f2d84f836b67-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.019593 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.019587    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b4371b-f550-4a21-8c82-f2d84f836b67-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.019593 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.019604    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.019832 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.019614    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ab6a659-425e-493a-b02d-9a1a23c06aa1-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.019832 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.019623    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwswq\" (UniqueName: \"kubernetes.io/projected/5ab6a659-425e-493a-b02d-9a1a23c06aa1-kube-api-access-zwswq\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.019832 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.019633    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-745w8\" (UniqueName: \"kubernetes.io/projected/f0b4371b-f550-4a21-8c82-f2d84f836b67-kube-api-access-745w8\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.019832 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.019642    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab6a659-425e-493a-b02d-9a1a23c06aa1-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.255147 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.255122    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 13:00:57.352715 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.352684    5663 generic.go:356] "Generic (PLEG): container finished" podID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerID="380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761" exitCode=0
Oct 31 13:00:57.352925 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.352745    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"
Oct 31 13:00:57.352925 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.352770    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" event={"ID":"5ab6a659-425e-493a-b02d-9a1a23c06aa1","Type":"ContainerDied","Data":"380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761"}
Oct 31 13:00:57.352925 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.352820    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch" event={"ID":"5ab6a659-425e-493a-b02d-9a1a23c06aa1","Type":"ContainerDied","Data":"b6cde3d6f8de4c78b88e79cf887b591c4ead319c10208a44a79c6f4d0f1a4ae9"}
Oct 31 13:00:57.352925 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.352847    5663 scope.go:122] "RemoveContainer" containerID="380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761"
Oct 31 13:00:57.354250 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.354230    5663 generic.go:356] "Generic (PLEG): container finished" podID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerID="a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918" exitCode=0
Oct 31 13:00:57.354412 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.354273    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"
Oct 31 13:00:57.354412 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.354322    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" event={"ID":"f0b4371b-f550-4a21-8c82-f2d84f836b67","Type":"ContainerDied","Data":"a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918"}
Oct 31 13:00:57.354412 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.354370    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" event={"ID":"f0b4371b-f550-4a21-8c82-f2d84f836b67","Type":"ContainerDied","Data":"aa232ee078a20e2d394158d4baec1254bfb4d8fe3d3fc6b52c0f4c6c6a16e8dd"}
Oct 31 13:00:57.355946 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.355928    5663 generic.go:356] "Generic (PLEG): container finished" podID="37639489-380a-4eab-b2c0-f1a61362d2db" containerID="8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c" exitCode=0
Oct 31 13:00:57.356060 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.355955    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" event={"ID":"37639489-380a-4eab-b2c0-f1a61362d2db","Type":"ContainerDied","Data":"8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c"}
Oct 31 13:00:57.356060 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.355971    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"
Oct 31 13:00:57.356060 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.356010    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn" event={"ID":"37639489-380a-4eab-b2c0-f1a61362d2db","Type":"ContainerDied","Data":"be5b520c83a4cee3ce41c35aac4686794904eca44cdd602691954741418741c2"}
Oct 31 13:00:57.366873 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.366859    5663 scope.go:122] "RemoveContainer" containerID="380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761"
Oct 31 13:00:57.367189 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:57.367141    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761\": container with ID starting with 380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761 not found: ID does not exist" containerID="380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761"
Oct 31 13:00:57.367189 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.367163    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761"} err="failed to get container status \"380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761\": rpc error: code = NotFound desc = could not find container \"380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761\": container with ID starting with 380cbb39389578000cc0d4b0cc9c8c9147caaf9b3158fee7687dd544c66f5761 not found: ID does not exist"
Oct 31 13:00:57.367189 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.367182    5663 scope.go:122] "RemoveContainer" containerID="a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918"
Oct 31 13:00:57.375866 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.375850    5663 scope.go:122] "RemoveContainer" containerID="a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918"
Oct 31 13:00:57.376097 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:57.376080    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918\": container with ID starting with a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918 not found: ID does not exist" containerID="a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918"
Oct 31 13:00:57.376145 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.376103    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918"} err="failed to get container status \"a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918\": rpc error: code = NotFound desc = could not find container \"a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918\": container with ID starting with a425641945b92c89a69046247c4780c443e6861b2fc5dfce301435b643863918 not found: ID does not exist"
Oct 31 13:00:57.376145 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.376117    5663 scope.go:122] "RemoveContainer" containerID="8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c"
Oct 31 13:00:57.384668 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.384646    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"]
Oct 31 13:00:57.385187 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.385161    5663 scope.go:122] "RemoveContainer" containerID="8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c"
Oct 31 13:00:57.386517 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:00:57.386496    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c\": container with ID starting with 8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c not found: ID does not exist" containerID="8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c"
Oct 31 13:00:57.386602 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.386522    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c"} err="failed to get container status \"8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c\": rpc error: code = NotFound desc = could not find container \"8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c\": container with ID starting with 8633a2508cb340cf616e9ad805ad5065e728192b61b8e162e5a7918ebf26e39c not found: ID does not exist"
Oct 31 13:00:57.387438 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.387420    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/simulator-w6-67d410-kserve-8588f46bcf-8b2ch"]
Oct 31 13:00:57.397843 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.397821    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"]
Oct 31 13:00:57.403935 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.403915    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq"]
Oct 31 13:00:57.423458 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423435    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-kube-api-access-q5tjx\" (UniqueName: \"kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-kube-api-access-q5tjx\") pod \"37639489-380a-4eab-b2c0-f1a61362d2db\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") "
Oct 31 13:00:57.423577 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423483    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-data\") pod \"37639489-380a-4eab-b2c0-f1a61362d2db\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") "
Oct 31 13:00:57.423577 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423512    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-istio-token\" (UniqueName: \"kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-istio-token\") pod \"37639489-380a-4eab-b2c0-f1a61362d2db\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") "
Oct 31 13:00:57.423664 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423635    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-certs\") pod \"37639489-380a-4eab-b2c0-f1a61362d2db\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") "
Oct 31 13:00:57.423709 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423670    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-credential-socket\") pod \"37639489-380a-4eab-b2c0-f1a61362d2db\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") "
Oct 31 13:00:57.423750 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423705    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-socket\") pod \"37639489-380a-4eab-b2c0-f1a61362d2db\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") "
Oct 31 13:00:57.423799 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423753    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37639489-380a-4eab-b2c0-f1a61362d2db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37639489-380a-4eab-b2c0-f1a61362d2db-istio-podinfo\") pod \"37639489-380a-4eab-b2c0-f1a61362d2db\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") "
Oct 31 13:00:57.423855 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423796    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-data" pod "37639489-380a-4eab-b2c0-f1a61362d2db" (UID: "37639489-380a-4eab-b2c0-f1a61362d2db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:57.423913 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423868    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-envoy\") pod \"37639489-380a-4eab-b2c0-f1a61362d2db\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") "
Oct 31 13:00:57.423972 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423908    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37639489-380a-4eab-b2c0-f1a61362d2db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37639489-380a-4eab-b2c0-f1a61362d2db-istiod-ca-cert\") pod \"37639489-380a-4eab-b2c0-f1a61362d2db\" (UID: \"37639489-380a-4eab-b2c0-f1a61362d2db\") "
Oct 31 13:00:57.424081 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423906    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-credential-socket" pod "37639489-380a-4eab-b2c0-f1a61362d2db" (UID: "37639489-380a-4eab-b2c0-f1a61362d2db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:57.424172 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423920    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-certs" pod "37639489-380a-4eab-b2c0-f1a61362d2db" (UID: "37639489-380a-4eab-b2c0-f1a61362d2db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:57.424172 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.423984    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-socket" pod "37639489-380a-4eab-b2c0-f1a61362d2db" (UID: "37639489-380a-4eab-b2c0-f1a61362d2db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:57.424290 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.424236    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37639489-380a-4eab-b2c0-f1a61362d2db-istiod-ca-cert" pod "37639489-380a-4eab-b2c0-f1a61362d2db" (UID: "37639489-380a-4eab-b2c0-f1a61362d2db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:57.424570 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.424549    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.424668 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.424575    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.424668 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.424592    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.424668 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.424607    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.424668 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.424622    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37639489-380a-4eab-b2c0-f1a61362d2db-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.425707 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.425686    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-kube-api-access-q5tjx" pod "37639489-380a-4eab-b2c0-f1a61362d2db" (UID: "37639489-380a-4eab-b2c0-f1a61362d2db"). InnerVolumeSpecName "kube-api-access-q5tjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:57.425707 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.425696    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-istio-token" pod "37639489-380a-4eab-b2c0-f1a61362d2db" (UID: "37639489-380a-4eab-b2c0-f1a61362d2db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:57.425856 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.425841    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-envoy" pod "37639489-380a-4eab-b2c0-f1a61362d2db" (UID: "37639489-380a-4eab-b2c0-f1a61362d2db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:57.425957 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.425943    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37639489-380a-4eab-b2c0-f1a61362d2db-istio-podinfo" pod "37639489-380a-4eab-b2c0-f1a61362d2db" (UID: "37639489-380a-4eab-b2c0-f1a61362d2db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:00:57.437506 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.437482    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" path="/var/lib/kubelet/pods/5ab6a659-425e-493a-b02d-9a1a23c06aa1/volumes"
Oct 31 13:00:57.437887 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.437874    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" path="/var/lib/kubelet/pods/f0b4371b-f550-4a21-8c82-f2d84f836b67/volumes"
Oct 31 13:00:57.526050 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.526018    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5tjx\" (UniqueName: \"kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-kube-api-access-q5tjx\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.526050 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.526046    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37639489-380a-4eab-b2c0-f1a61362d2db-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.526050 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.526056    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37639489-380a-4eab-b2c0-f1a61362d2db-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.526530 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.526067    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37639489-380a-4eab-b2c0-f1a61362d2db-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:57.672190 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.672115    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"]
Oct 31 13:00:57.678645 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.678620    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c67de2-openshift-default-6d9489bd55-xwvwn"]
Oct 31 13:00:57.767445 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:57.767403    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-67d410/distinct-2-w6-67d410-kserve-7748c5c886-xbqwq" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 13:00:59.441454 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:00:59.441417    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" path="/var/lib/kubelet/pods/37639489-380a-4eab-b2c0-f1a61362d2db/volumes"
Oct 31 13:01:02.404716 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.404627    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"]
Oct 31 13:01:02.405183 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.404850    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb" podUID="96e5a20b-4141-4cbe-89dd-888c80811889" containerName="maas-api" containerID="cri-o://c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42" gracePeriod=30
Oct 31 13:01:02.656536 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.656470    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 13:01:02.682427 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.682376    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e5a20b-4141-4cbe-89dd-888c80811889-kube-api-access-5dkkm\" (UniqueName: \"kubernetes.io/projected/96e5a20b-4141-4cbe-89dd-888c80811889-kube-api-access-5dkkm\") pod \"96e5a20b-4141-4cbe-89dd-888c80811889\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") "
Oct 31 13:01:02.686632 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.686091    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e5a20b-4141-4cbe-89dd-888c80811889-kube-api-access-5dkkm" pod "96e5a20b-4141-4cbe-89dd-888c80811889" (UID: "96e5a20b-4141-4cbe-89dd-888c80811889"). InnerVolumeSpecName "kube-api-access-5dkkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:01:02.783074 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.783034    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-metrics-certs\") pod \"96e5a20b-4141-4cbe-89dd-888c80811889\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") "
Oct 31 13:01:02.783074 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.783081    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-maas-api-tls\") pod \"96e5a20b-4141-4cbe-89dd-888c80811889\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") "
Oct 31 13:01:02.783384 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.783124    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e5a20b-4141-4cbe-89dd-888c80811889-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e5a20b-4141-4cbe-89dd-888c80811889-openshift-service-ca\") pod \"96e5a20b-4141-4cbe-89dd-888c80811889\" (UID: \"96e5a20b-4141-4cbe-89dd-888c80811889\") "
Oct 31 13:01:02.783485 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.783407    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dkkm\" (UniqueName: \"kubernetes.io/projected/96e5a20b-4141-4cbe-89dd-888c80811889-kube-api-access-5dkkm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:02.784550 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.784519    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e5a20b-4141-4cbe-89dd-888c80811889-openshift-service-ca" pod "96e5a20b-4141-4cbe-89dd-888c80811889" (UID: "96e5a20b-4141-4cbe-89dd-888c80811889"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:01:02.787884 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.787840    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-maas-api-tls" pod "96e5a20b-4141-4cbe-89dd-888c80811889" (UID: "96e5a20b-4141-4cbe-89dd-888c80811889"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:01:02.789815 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.789792    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-metrics-certs" pod "96e5a20b-4141-4cbe-89dd-888c80811889" (UID: "96e5a20b-4141-4cbe-89dd-888c80811889"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:01:02.884406 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.884341    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:02.884406 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.884406    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96e5a20b-4141-4cbe-89dd-888c80811889-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:02.884630 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:02.884419    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e5a20b-4141-4cbe-89dd-888c80811889-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:03.394867 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:03.394823    5663 generic.go:356] "Generic (PLEG): container finished" podID="96e5a20b-4141-4cbe-89dd-888c80811889" containerID="c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42" exitCode=0
Oct 31 13:01:03.395083 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:03.394889    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"
Oct 31 13:01:03.395083 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:03.394904    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb" event={"ID":"96e5a20b-4141-4cbe-89dd-888c80811889","Type":"ContainerDied","Data":"c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42"}
Oct 31 13:01:03.395083 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:03.394964    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb" event={"ID":"96e5a20b-4141-4cbe-89dd-888c80811889","Type":"ContainerDied","Data":"52f91cffce272050755638ed7af01dc6cd85f53c444bd01db8b5ca7c5711985c"}
Oct 31 13:01:03.395083 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:03.394979    5663 scope.go:122] "RemoveContainer" containerID="c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42"
Oct 31 13:01:03.406702 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:03.406412    5663 scope.go:122] "RemoveContainer" containerID="c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42"
Oct 31 13:01:03.407931 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:01:03.407901    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42\": container with ID starting with c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42 not found: ID does not exist" containerID="c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42"
Oct 31 13:01:03.408026 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:03.407943    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42"} err="failed to get container status \"c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42\": rpc error: code = NotFound desc = could not find container \"c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42\": container with ID starting with c11f544f080b7e1d159440438de83a81c65b405725c965b91d072ae3ba370c42 not found: ID does not exist"
Oct 31 13:01:03.422830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:03.422802    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"]
Oct 31 13:01:03.425392 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:03.425367    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-67d410-5864f5fd5-jjqlb"]
Oct 31 13:01:03.436971 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:03.436943    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e5a20b-4141-4cbe-89dd-888c80811889" path="/var/lib/kubelet/pods/96e5a20b-4141-4cbe-89dd-888c80811889/volumes"
Oct 31 13:01:09.464288 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:09.464249    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"]
Oct 31 13:01:09.464839 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:09.464542    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerName="istio-proxy" containerID="cri-o://8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7" gracePeriod=30
Oct 31 13:01:14.750661 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.750605    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 13:01:14.792377 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.792297    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-token\" (UniqueName: \"kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-token\") pod \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") "
Oct 31 13:01:14.792377 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.792366    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-podinfo\") pod \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") "
Oct 31 13:01:14.792621 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.792443    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-kube-api-access-ljzfz\" (UniqueName: \"kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-kube-api-access-ljzfz\") pod \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") "
Oct 31 13:01:14.792621 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.792482    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-data\") pod \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") "
Oct 31 13:01:14.792621 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.792551    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-socket\") pod \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") "
Oct 31 13:01:14.792621 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.792590    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istiod-ca-cert\") pod \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") "
Oct 31 13:01:14.792621 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.792620    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-certs\") pod \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") "
Oct 31 13:01:14.792899 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.792661    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-envoy\") pod \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") "
Oct 31 13:01:14.792899 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.792691    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-credential-socket\") pod \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\" (UID: \"b163c8ec-22bc-4195-a75b-a9e5602c69ad\") "
Oct 31 13:01:14.813383 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.795321    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-socket" pod "b163c8ec-22bc-4195-a75b-a9e5602c69ad" (UID: "b163c8ec-22bc-4195-a75b-a9e5602c69ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:01:14.813383 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.795963    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-credential-socket" pod "b163c8ec-22bc-4195-a75b-a9e5602c69ad" (UID: "b163c8ec-22bc-4195-a75b-a9e5602c69ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:01:14.813383 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.797038    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-certs" pod "b163c8ec-22bc-4195-a75b-a9e5602c69ad" (UID: "b163c8ec-22bc-4195-a75b-a9e5602c69ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:01:14.813383 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.797439    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istiod-ca-cert" pod "b163c8ec-22bc-4195-a75b-a9e5602c69ad" (UID: "b163c8ec-22bc-4195-a75b-a9e5602c69ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:01:14.813383 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.798619    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-data" pod "b163c8ec-22bc-4195-a75b-a9e5602c69ad" (UID: "b163c8ec-22bc-4195-a75b-a9e5602c69ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:01:14.823760 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.823694    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-token" pod "b163c8ec-22bc-4195-a75b-a9e5602c69ad" (UID: "b163c8ec-22bc-4195-a75b-a9e5602c69ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:01:14.823904 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.823778    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-envoy" pod "b163c8ec-22bc-4195-a75b-a9e5602c69ad" (UID: "b163c8ec-22bc-4195-a75b-a9e5602c69ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:01:14.824213 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.824173    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-kube-api-access-ljzfz" pod "b163c8ec-22bc-4195-a75b-a9e5602c69ad" (UID: "b163c8ec-22bc-4195-a75b-a9e5602c69ad"). InnerVolumeSpecName "kube-api-access-ljzfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:01:14.824621 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.824596    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-podinfo" pod "b163c8ec-22bc-4195-a75b-a9e5602c69ad" (UID: "b163c8ec-22bc-4195-a75b-a9e5602c69ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:01:14.894433 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.894392    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljzfz\" (UniqueName: \"kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-kube-api-access-ljzfz\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:14.894433 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.894431    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:14.894695 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.894446    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:14.894695 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.894462    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:14.894695 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.894476    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:14.894695 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.894489    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:14.894695 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.894502    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b163c8ec-22bc-4195-a75b-a9e5602c69ad-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:14.894695 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.894516    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:14.894695 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:14.894529    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b163c8ec-22bc-4195-a75b-a9e5602c69ad-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:15.463284 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:15.463246    5663 generic.go:356] "Generic (PLEG): container finished" podID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerID="8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7" exitCode=0
Oct 31 13:01:15.463459 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:15.463309    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"
Oct 31 13:01:15.463459 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:15.463412    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" event={"ID":"b163c8ec-22bc-4195-a75b-a9e5602c69ad","Type":"ContainerDied","Data":"8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7"}
Oct 31 13:01:15.463597 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:15.463464    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb" event={"ID":"b163c8ec-22bc-4195-a75b-a9e5602c69ad","Type":"ContainerDied","Data":"f98a18b5f6eccf744917821404643a9ca7a86ee2aa0a2d1b1d56a3e669b875a3"}
Oct 31 13:01:15.463597 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:15.463494    5663 scope.go:122] "RemoveContainer" containerID="8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7"
Oct 31 13:01:15.479910 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:15.479854    5663 scope.go:122] "RemoveContainer" containerID="8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7"
Oct 31 13:01:15.480986 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:01:15.480923    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7\": container with ID starting with 8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7 not found: ID does not exist" containerID="8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7"
Oct 31 13:01:15.481098 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:15.480994    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7"} err="failed to get container status \"8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7\": rpc error: code = NotFound desc = could not find container \"8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7\": container with ID starting with 8daf2e9bfee8d3d7d594daaf4e72ee2a608f199bddce1732a65dc74e0faf88e7 not found: ID does not exist"
Oct 31 13:01:15.498464 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:15.492428    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"]
Oct 31 13:01:15.500920 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:15.500894    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-67d410-openshift-default-676b978549-8dtnb"]
Oct 31 13:01:17.436936 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:17.436897    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" path="/var/lib/kubelet/pods/b163c8ec-22bc-4195-a75b-a9e5602c69ad/volumes"
Oct 31 13:01:22.475741 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:01:22.475706    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:01:47.150559 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.150528    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"]
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.150948    5663 cpu_manager.go:414] "containerMap: removing container" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.150963    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.150980    5663 cpu_manager.go:414] "containerMap: removing container" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" containerName="istio-proxy"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.150985    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" containerName="istio-proxy"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.150993    5663 cpu_manager.go:414] "containerMap: removing container" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerName="istio-proxy"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151000    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerName="istio-proxy"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151023    5663 cpu_manager.go:414] "containerMap: removing container" podUID="e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" containerName="istio-proxy"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151029    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bdfa24-6e4e-4d2c-a4ac-2c515e104761" containerName="istio-proxy"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151035    5663 cpu_manager.go:414] "containerMap: removing container" podUID="96e5a20b-4141-4cbe-89dd-888c80811889" containerName="maas-api"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151039    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e5a20b-4141-4cbe-89dd-888c80811889" containerName="maas-api"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151053    5663 cpu_manager.go:414] "containerMap: removing container" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151057    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151066    5663 cpu_manager.go:414] "containerMap: removing container" podUID="a79edd95-23ec-420c-9fae-73f2623175fd" containerName="revoke-keys"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151071    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79edd95-23ec-420c-9fae-73f2623175fd" containerName="revoke-keys"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151131    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e5a20b-4141-4cbe-89dd-888c80811889" containerName="maas-api"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151145    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab6a659-425e-493a-b02d-9a1a23c06aa1" containerName="main"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151154    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79edd95-23ec-420c-9fae-73f2623175fd" containerName="revoke-keys"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151162    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b4371b-f550-4a21-8c82-f2d84f836b67" containerName="main"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151172    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="37639489-380a-4eab-b2c0-f1a61362d2db" containerName="istio-proxy"
Oct 31 13:01:47.151270 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.151179    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="b163c8ec-22bc-4195-a75b-a9e5602c69ad" containerName="istio-proxy"
Oct 31 13:01:47.154597 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.154577    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.176648 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.176616    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"]
Oct 31 13:01:47.284463 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.284428    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.284463 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.284469    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.284680 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.284490    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.284680 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.284625    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsjt5\" (UniqueName: \"kubernetes.io/projected/c8b74799-9fa1-41b6-99b1-6dc9a656841d-kube-api-access-rsjt5\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.284680 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.284657    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.284782 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.284720    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.284782 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.284747    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.284782 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.284768    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.284878 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.284801    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.385228 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.385188    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsjt5\" (UniqueName: \"kubernetes.io/projected/c8b74799-9fa1-41b6-99b1-6dc9a656841d-kube-api-access-rsjt5\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.385228 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.385229    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.385523 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.385264    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.385523 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.385294    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.385523 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.385316    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.385523 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.385386    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.385523 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.385460    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.385523 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.385495    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.385796 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.385525    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.385833 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.385805    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.385892 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.385837    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.385991 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.385969    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.386101 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.386082    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.386181 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.386154    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.387935 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.387913    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.388539 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.388520    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.397729 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.397698    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b74799-9fa1-41b6-99b1-6dc9a656841d-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.397850 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.397836    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsjt5\" (UniqueName: \"kubernetes.io/projected/c8b74799-9fa1-41b6-99b1-6dc9a656841d-kube-api-access-rsjt5\") pod \"maas-default-gateway-openshift-default-645cf464d7-7q2ln\" (UID: \"c8b74799-9fa1-41b6-99b1-6dc9a656841d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.469271 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.469171    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:47.746914 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.746891    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"]
Oct 31 13:01:47.749603 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:01:47.749574    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b74799_9fa1_41b6_99b1_6dc9a656841d.slice/crio-ea2ce23b0b54b713ace9fb8849ea6df4acf788dca1de0c803bcf5487af241347 WatchSource:0}: Error finding container ea2ce23b0b54b713ace9fb8849ea6df4acf788dca1de0c803bcf5487af241347: Status 404 returned error can't find the container with id ea2ce23b0b54b713ace9fb8849ea6df4acf788dca1de0c803bcf5487af241347
Oct 31 13:01:47.751623 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.751587    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:01:47.751683 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.751659    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:01:47.751723 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:47.751691    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:01:48.617339 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:48.617304    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln" event={"ID":"c8b74799-9fa1-41b6-99b1-6dc9a656841d","Type":"ContainerStarted","Data":"bacbcd23dffcfaedb8082817cc2a10188393702b2bccafa7bbb95eaecc28fb9a"}
Oct 31 13:01:48.617755 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:48.617376    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln" event={"ID":"c8b74799-9fa1-41b6-99b1-6dc9a656841d","Type":"ContainerStarted","Data":"ea2ce23b0b54b713ace9fb8849ea6df4acf788dca1de0c803bcf5487af241347"}
Oct 31 13:01:48.640838 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:48.640788    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln" podStartSLOduration=1.640773335 podStartE2EDuration="1.640773335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:01:48.639664332 +0000 UTC m=+2033.380090560" watchObservedRunningTime="2026-10-31 13:01:48.640773335 +0000 UTC m=+2033.381199551"
Oct 31 13:01:49.475334 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:49.475300    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:49.475741 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:49.475719    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7q2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 13:01:49.475937 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:49.475912    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln" podUID="c8b74799-9fa1-41b6-99b1-6dc9a656841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 13:01:50.476115 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:50.474465    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7q2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 13:01:50.476115 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:50.474530    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln" podUID="c8b74799-9fa1-41b6-99b1-6dc9a656841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 13:01:51.474821 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:51.474781    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7q2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 13:01:51.475137 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:51.475105    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln" podUID="c8b74799-9fa1-41b6-99b1-6dc9a656841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 13:01:52.470844 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:52.470804    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7q2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 13:01:52.471450 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:52.471413    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln" podUID="c8b74799-9fa1-41b6-99b1-6dc9a656841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 13:01:52.499468 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:01:52.499434    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:01:53.475874 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:53.475838    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7q2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 13:01:53.476989 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:53.476954    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln" podUID="c8b74799-9fa1-41b6-99b1-6dc9a656841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 13:01:54.022174 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:54.022136    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"]
Oct 31 13:01:54.025256 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:54.025226    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9badbf0-p6ljb"]
Oct 31 13:01:54.470264 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:54.470225    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-7q2ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 13:01:54.470499 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:54.470298    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln" podUID="c8b74799-9fa1-41b6-99b1-6dc9a656841d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 13:01:55.439072 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:55.439036    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d987d0-d800-436e-b59e-779b730a976e" path="/var/lib/kubelet/pods/57d987d0-d800-436e-b59e-779b730a976e/volumes"
Oct 31 13:01:55.880861 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:55.880829    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:55.881234 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:55.881206    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:01:55.882007 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:01:55.881985    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-7q2ln"
Oct 31 13:02:22.477090 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:02:22.477045    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:02:52.475098 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:02:52.475020    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:02:59.519849 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:02:59.519813    5663 scope.go:122] "RemoveContainer" containerID="f3f567526c477d0d9aef807b3fa20c5c2345389a57602dc97f5da663e708c6b9"
Oct 31 13:02:59.528602 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:02:59.528580    5663 scope.go:122] "RemoveContainer" containerID="b965834faeee4d9f24f1f99a5f504b4330318a5a5629e60fd97ae7b67ef5d7e0"
Oct 31 13:03:22.475318 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:03:22.475280    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:03:25.025535 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:03:25.025495    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"]
Oct 31 13:03:25.028130 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:03:25.028097    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69b0a28-86pgd"]
Oct 31 13:03:25.030894 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:03:25.030870    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"]
Oct 31 13:03:25.033026 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:03:25.033005    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-170f8916-qmw92"]
Oct 31 13:03:25.436933 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:03:25.436896    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e72666-f930-4fc9-a5b7-7d42f56b9d83" path="/var/lib/kubelet/pods/21e72666-f930-4fc9-a5b7-7d42f56b9d83/volumes"
Oct 31 13:03:25.437314 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:03:25.437300    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e597fa9f-7e3e-40b2-b841-e4a7d7bed491" path="/var/lib/kubelet/pods/e597fa9f-7e3e-40b2-b841-e4a7d7bed491/volumes"
Oct 31 13:03:37.026170 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:03:37.025969    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"]
Oct 31 13:03:37.031750 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:03:37.031718    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743d23d0-5c4vt"]
Oct 31 13:03:37.440263 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:03:37.440222    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c66a3a3-5c0b-4c44-a3d6-f80149fc4244" path="/var/lib/kubelet/pods/8c66a3a3-5c0b-4c44-a3d6-f80149fc4244/volumes"
Oct 31 13:03:39.038242 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:03:39.038203    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"]
Oct 31 13:03:39.042937 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:03:39.042900    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4ce7a1-22wwh"]
Oct 31 13:03:39.436492 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:03:39.436452    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6" path="/var/lib/kubelet/pods/0b1abbd9-1ed5-4842-87bb-ff328bb9d7a6/volumes"
Oct 31 13:03:52.475660 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:03:52.475616    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:04:10.863380 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:04:10.860543    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-dswld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:04:10.863380 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:04:10.860632    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" podUID="57ad591e-3b45-4312-8612-cf848dcf55e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:04:22.475170 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:04:22.475127    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:04:48.033584 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:04:48.033543    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"]
Oct 31 13:04:48.036797 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:04:48.036762    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce73be33-pgw6k"]
Oct 31 13:04:49.436815 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:04:49.436778    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69e066f-edff-4419-8975-bac11fe1a8f3" path="/var/lib/kubelet/pods/d69e066f-edff-4419-8975-bac11fe1a8f3/volumes"
Oct 31 13:04:52.476395 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:04:52.476339    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:04:59.558922 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:04:59.558889    5663 scope.go:122] "RemoveContainer" containerID="ac606627a28f5f7bf827cb0f9461e405b7b91479ecacdb37a5acb8e8195d5b6c"
Oct 31 13:04:59.569265 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:04:59.569229    5663 scope.go:122] "RemoveContainer" containerID="6eeae6ba8fcbc88e0fba030fe75e2623d4f74c6472d3dd5517e923d93f8ad196"
Oct 31 13:04:59.577306 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:04:59.577283    5663 scope.go:122] "RemoveContainer" containerID="1835b6244c5489ff9e4cde31b651b1dd12e8882d21314342da2dd459e8a65537"
Oct 31 13:04:59.585181 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:04:59.585156    5663 scope.go:122] "RemoveContainer" containerID="6c2224460d4ea139199f31b25ce189cd071ddabed00023d8468d2e4fbfc51518"
Oct 31 13:04:59.592666 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:04:59.592647    5663 scope.go:122] "RemoveContainer" containerID="23d2ddcb8e8a1ae6cceb16b8228977f923431aae2807c18b1e524c25ebebfa68"
Oct 31 13:04:59.599881 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:04:59.599861    5663 scope.go:122] "RemoveContainer" containerID="0864ba6e3b1e9560548f088f94b613894c691df4f453d0bc38e0d22862124b83"
Oct 31 13:05:22.475393 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:05:22.475327    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:05:25.934379 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:25.934323    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"]
Oct 31 13:05:25.936857 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:25.934531    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs" podUID="081a90a7-3aa0-47c6-b528-c103b3a8dc62" containerName="discovery" containerID="cri-o://38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a" gracePeriod=30
Oct 31 13:05:26.180455 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.180429    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 13:05:26.225750 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.225666    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081a90a7-3aa0-47c6-b528-c103b3a8dc62-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/081a90a7-3aa0-47c6-b528-c103b3a8dc62-local-certs\") pod \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") "
Oct 31 13:05:26.225750 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.225715    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-kube-api-access-kn84g\" (UniqueName: \"kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-kube-api-access-kn84g\") pod \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") "
Oct 31 13:05:26.225961 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.225779    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-token\" (UniqueName: \"kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-token\") pod \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") "
Oct 31 13:05:26.225961 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.225830    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-kubeconfig\") pod \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") "
Oct 31 13:05:26.225961 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.225883    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-dns-cert\") pod \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") "
Oct 31 13:05:26.225961 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.225930    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-ca-configmap\") pod \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") "
Oct 31 13:05:26.226196 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.225963    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-cacerts\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-cacerts\") pod \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\" (UID: \"081a90a7-3aa0-47c6-b528-c103b3a8dc62\") "
Oct 31 13:05:26.226544 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.226493    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-ca-configmap" pod "081a90a7-3aa0-47c6-b528-c103b3a8dc62" (UID: "081a90a7-3aa0-47c6-b528-c103b3a8dc62"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:05:26.228632 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.228604    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081a90a7-3aa0-47c6-b528-c103b3a8dc62-local-certs" pod "081a90a7-3aa0-47c6-b528-c103b3a8dc62" (UID: "081a90a7-3aa0-47c6-b528-c103b3a8dc62"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:05:26.228761 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.228639    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-token" pod "081a90a7-3aa0-47c6-b528-c103b3a8dc62" (UID: "081a90a7-3aa0-47c6-b528-c103b3a8dc62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:05:26.228761 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.228725    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-kube-api-access-kn84g" pod "081a90a7-3aa0-47c6-b528-c103b3a8dc62" (UID: "081a90a7-3aa0-47c6-b528-c103b3a8dc62"). InnerVolumeSpecName "kube-api-access-kn84g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:05:26.228883 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.228830    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-kubeconfig" pod "081a90a7-3aa0-47c6-b528-c103b3a8dc62" (UID: "081a90a7-3aa0-47c6-b528-c103b3a8dc62"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:05:26.229213 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.229181    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-dns-cert" pod "081a90a7-3aa0-47c6-b528-c103b3a8dc62" (UID: "081a90a7-3aa0-47c6-b528-c103b3a8dc62"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:05:26.229336 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.229225    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-cacerts" pod "081a90a7-3aa0-47c6-b528-c103b3a8dc62" (UID: "081a90a7-3aa0-47c6-b528-c103b3a8dc62"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:05:26.327263 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.327231    5663 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/081a90a7-3aa0-47c6-b528-c103b3a8dc62-local-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:26.327263 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.327261    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn84g\" (UniqueName: \"kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-kube-api-access-kn84g\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:26.327263 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.327272    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:26.327538 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.327280    5663 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-kubeconfig\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:26.327538 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.327289    5663 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-dns-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:26.327538 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.327299    5663 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/081a90a7-3aa0-47c6-b528-c103b3a8dc62-istio-csr-ca-configmap\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:26.327538 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.327309    5663 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/081a90a7-3aa0-47c6-b528-c103b3a8dc62-cacerts\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:26.818500 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.818462    5663 generic.go:356] "Generic (PLEG): container finished" podID="081a90a7-3aa0-47c6-b528-c103b3a8dc62" containerID="38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a" exitCode=0
Oct 31 13:05:26.818690 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.818513    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"
Oct 31 13:05:26.818690 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.818545    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs" event={"ID":"081a90a7-3aa0-47c6-b528-c103b3a8dc62","Type":"ContainerDied","Data":"38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a"}
Oct 31 13:05:26.818690 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.818600    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs" event={"ID":"081a90a7-3aa0-47c6-b528-c103b3a8dc62","Type":"ContainerDied","Data":"2c35a067e311db3a2d2d8e3061cb0a02042208251fd1c71fc742ef622ac7a705"}
Oct 31 13:05:26.818690 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.818620    5663 scope.go:122] "RemoveContainer" containerID="38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a"
Oct 31 13:05:26.835452 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.835432    5663 scope.go:122] "RemoveContainer" containerID="38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a"
Oct 31 13:05:26.835746 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:05:26.835727    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a\": container with ID starting with 38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a not found: ID does not exist" containerID="38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a"
Oct 31 13:05:26.835796 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.835757    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a"} err="failed to get container status \"38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a\": rpc error: code = NotFound desc = could not find container \"38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a\": container with ID starting with 38f2c3ae5c0b0845616ac4c13f99b694a07d6fdd1fdfd0bda7397896c40ff39a not found: ID does not exist"
Oct 31 13:05:26.840167 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.840144    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"]
Oct 31 13:05:26.851000 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:26.850977    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4ggxs"]
Oct 31 13:05:27.437057 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:27.437022    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081a90a7-3aa0-47c6-b528-c103b3a8dc62" path="/var/lib/kubelet/pods/081a90a7-3aa0-47c6-b528-c103b3a8dc62/volumes"
Oct 31 13:05:33.028790 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:33.028693    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"]
Oct 31 13:05:33.036271 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:33.036242    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603e53c1-b9rpn"]
Oct 31 13:05:33.438783 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:33.438738    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422a0516-aa07-47c6-9acd-e56123690393" path="/var/lib/kubelet/pods/422a0516-aa07-47c6-9acd-e56123690393/volumes"
Oct 31 13:05:50.003006 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.002967    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"]
Oct 31 13:05:50.003454 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.003313    5663 cpu_manager.go:414] "containerMap: removing container" podUID="081a90a7-3aa0-47c6-b528-c103b3a8dc62" containerName="discovery"
Oct 31 13:05:50.003454 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.003323    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081a90a7-3aa0-47c6-b528-c103b3a8dc62" containerName="discovery"
Oct 31 13:05:50.003454 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.003392    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="081a90a7-3aa0-47c6-b528-c103b3a8dc62" containerName="discovery"
Oct 31 13:05:50.006171 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.006154    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:50.011388 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.011366    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:05:50.011531 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.011448    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:05:50.016962 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.016932    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"]
Oct 31 13:05:50.072662 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.072629    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-lhvnx\" (UID: \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:50.072817 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.072725    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-lhvnx\" (UID: \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:50.072817 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.072787    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn4fb\" (UniqueName: \"kubernetes.io/projected/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-kube-api-access-mn4fb\") pod \"maas-api-revoke-sub-e2fc9c5b-lhvnx\" (UID: \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:50.173195 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.173163    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-lhvnx\" (UID: \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:50.173405 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.173208    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn4fb\" (UniqueName: \"kubernetes.io/projected/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-kube-api-access-mn4fb\") pod \"maas-api-revoke-sub-e2fc9c5b-lhvnx\" (UID: \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:50.173405 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.173242    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-lhvnx\" (UID: \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:50.173828 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.173807    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-lhvnx\" (UID: \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:50.175532 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.175513    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-lhvnx\" (UID: \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:50.185874 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.185854    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn4fb\" (UniqueName: \"kubernetes.io/projected/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-kube-api-access-mn4fb\") pod \"maas-api-revoke-sub-e2fc9c5b-lhvnx\" (UID: \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:50.319398 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.319372    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:50.594836 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.594765    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"]
Oct 31 13:05:50.597621 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:05:50.597591    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87f0d5b_4dc2_4f2a_bc0f_428540f395e4.slice/crio-846902cbddc40693b90232ede5f87d756cde0c4df341734163677d4b65beb582 WatchSource:0}: Error finding container 846902cbddc40693b90232ede5f87d756cde0c4df341734163677d4b65beb582: Status 404 returned error can't find the container with id 846902cbddc40693b90232ede5f87d756cde0c4df341734163677d4b65beb582
Oct 31 13:05:50.930449 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.930414    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx" event={"ID":"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4","Type":"ContainerStarted","Data":"efa0bc577f846034757a3a5b9e0fa09d58845b6c78238a5435cd4456dd33789c"}
Oct 31 13:05:50.930615 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:50.930463    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx" event={"ID":"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4","Type":"ContainerStarted","Data":"846902cbddc40693b90232ede5f87d756cde0c4df341734163677d4b65beb582"}
Oct 31 13:05:51.026451 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:51.026397    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx" podStartSLOduration=2.026382832 podStartE2EDuration="2.026382832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:50.952830078 +0000 UTC m=+2275.693256296" watchObservedRunningTime="2026-10-31 13:05:51.026382832 +0000 UTC m=+2275.766809047"
Oct 31 13:05:51.026846 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:51.026627    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"]
Oct 31 13:05:51.031829 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:51.031806    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98638026-n4f27"]
Oct 31 13:05:51.438068 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:51.438038    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79edd95-23ec-420c-9fae-73f2623175fd" path="/var/lib/kubelet/pods/a79edd95-23ec-420c-9fae-73f2623175fd/volumes"
Oct 31 13:05:51.936506 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:51.936472    5663 generic.go:356] "Generic (PLEG): container finished" podID="b87f0d5b-4dc2-4f2a-bc0f-428540f395e4" containerID="efa0bc577f846034757a3a5b9e0fa09d58845b6c78238a5435cd4456dd33789c" exitCode=0
Oct 31 13:05:51.936684 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:51.936512    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx" event={"ID":"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4","Type":"ContainerDied","Data":"efa0bc577f846034757a3a5b9e0fa09d58845b6c78238a5435cd4456dd33789c"}
Oct 31 13:05:52.475190 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:05:52.475154    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:05:53.079196 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.079167    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:53.093752 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.093704    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-kube-api-access-mn4fb\" (UniqueName: \"kubernetes.io/projected/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-kube-api-access-mn4fb\") pod \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\" (UID: \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\") "
Oct 31 13:05:53.093752 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.093755    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-openshift-ca\") pod \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\" (UID: \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\") "
Oct 31 13:05:53.094044 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.093799    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-xks-ca\") pod \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\" (UID: \"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4\") "
Oct 31 13:05:53.094191 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.094164    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-openshift-ca" pod "b87f0d5b-4dc2-4f2a-bc0f-428540f395e4" (UID: "b87f0d5b-4dc2-4f2a-bc0f-428540f395e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:05:53.095907 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.095879    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-xks-ca" pod "b87f0d5b-4dc2-4f2a-bc0f-428540f395e4" (UID: "b87f0d5b-4dc2-4f2a-bc0f-428540f395e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:05:53.096032 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.095971    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-kube-api-access-mn4fb" pod "b87f0d5b-4dc2-4f2a-bc0f-428540f395e4" (UID: "b87f0d5b-4dc2-4f2a-bc0f-428540f395e4"). InnerVolumeSpecName "kube-api-access-mn4fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:05:53.194485 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.194413    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn4fb\" (UniqueName: \"kubernetes.io/projected/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-kube-api-access-mn4fb\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:53.194485 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.194441    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:53.194485 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.194452    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:05:53.946326 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.946291    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx" event={"ID":"b87f0d5b-4dc2-4f2a-bc0f-428540f395e4","Type":"ContainerDied","Data":"846902cbddc40693b90232ede5f87d756cde0c4df341734163677d4b65beb582"}
Oct 31 13:05:53.946326 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.946322    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"
Oct 31 13:05:53.946827 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:53.946340    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="846902cbddc40693b90232ede5f87d756cde0c4df341734163677d4b65beb582"
Oct 31 13:05:59.633441 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:59.633406    5663 scope.go:122] "RemoveContainer" containerID="1de6b3211da58618a43160f58bae1ecab6a9f2806ace12090c93451c8141807e"
Oct 31 13:05:59.642115 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:05:59.642091    5663 scope.go:122] "RemoveContainer" containerID="9ed9d8391d488b4e777a447247aa5d4c9b474810ebe6fd0bb93399c743185282"
Oct 31 13:06:00.232061 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.232025    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"]
Oct 31 13:06:00.232437 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.232422    5663 cpu_manager.go:414] "containerMap: removing container" podUID="b87f0d5b-4dc2-4f2a-bc0f-428540f395e4" containerName="revoke-keys"
Oct 31 13:06:00.232501 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.232438    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b87f0d5b-4dc2-4f2a-bc0f-428540f395e4" containerName="revoke-keys"
Oct 31 13:06:00.232539 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.232519    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="b87f0d5b-4dc2-4f2a-bc0f-428540f395e4" containerName="revoke-keys"
Oct 31 13:06:00.237107 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.237091    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:00.242841 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.242806    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:06:00.242841 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.243071    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:06:00.246239 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.246214    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"]
Oct 31 13:06:00.258932 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.258903    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9g4h2\" (UID: \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:00.259091 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.258950    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9g4h2\" (UID: \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:00.259091 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.259047    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28msj\" (UniqueName: \"kubernetes.io/projected/36fc0fe4-fd68-4340-b90e-e739b4b3534e-kube-api-access-28msj\") pod \"maas-api-revoke-sub-5a29efb3-9g4h2\" (UID: \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:00.360203 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.360164    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9g4h2\" (UID: \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:00.360435 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.360215    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9g4h2\" (UID: \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:00.360435 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.360271    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28msj\" (UniqueName: \"kubernetes.io/projected/36fc0fe4-fd68-4340-b90e-e739b4b3534e-kube-api-access-28msj\") pod \"maas-api-revoke-sub-5a29efb3-9g4h2\" (UID: \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:00.360945 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.360926    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9g4h2\" (UID: \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:00.362593 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.362565    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9g4h2\" (UID: \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:00.375341 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.375316    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28msj\" (UniqueName: \"kubernetes.io/projected/36fc0fe4-fd68-4340-b90e-e739b4b3534e-kube-api-access-28msj\") pod \"maas-api-revoke-sub-5a29efb3-9g4h2\" (UID: \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:00.553522 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.553431    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:00.832108 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.832077    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"]
Oct 31 13:06:00.835051 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:06:00.835021    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fc0fe4_fd68_4340_b90e_e739b4b3534e.slice/crio-61d12e25be9decaf8b8743f94563d2ff26ea5d89d89dca74fcb403c6b440c6ce WatchSource:0}: Error finding container 61d12e25be9decaf8b8743f94563d2ff26ea5d89d89dca74fcb403c6b440c6ce: Status 404 returned error can't find the container with id 61d12e25be9decaf8b8743f94563d2ff26ea5d89d89dca74fcb403c6b440c6ce
Oct 31 13:06:00.977075 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:00.977040    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2" event={"ID":"36fc0fe4-fd68-4340-b90e-e739b4b3534e","Type":"ContainerStarted","Data":"61d12e25be9decaf8b8743f94563d2ff26ea5d89d89dca74fcb403c6b440c6ce"}
Oct 31 13:06:01.985110 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:01.984970    5663 generic.go:356] "Generic (PLEG): container finished" podID="36fc0fe4-fd68-4340-b90e-e739b4b3534e" containerID="3970cff077d4d2c88a74f42cba5fce3db543b87703f653757290dcf206f8a38f" exitCode=0
Oct 31 13:06:01.985110 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:01.985028    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2" event={"ID":"36fc0fe4-fd68-4340-b90e-e739b4b3534e","Type":"ContainerDied","Data":"3970cff077d4d2c88a74f42cba5fce3db543b87703f653757290dcf206f8a38f"}
Oct 31 13:06:03.322377 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:03.321743    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:03.431382 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:03.431056    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-openshift-ca\") pod \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\" (UID: \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\") "
Oct 31 13:06:03.431382 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:03.431174    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36fc0fe4-fd68-4340-b90e-e739b4b3534e-kube-api-access-28msj\" (UniqueName: \"kubernetes.io/projected/36fc0fe4-fd68-4340-b90e-e739b4b3534e-kube-api-access-28msj\") pod \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\" (UID: \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\") "
Oct 31 13:06:03.431382 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:03.431249    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-xks-ca\") pod \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\" (UID: \"36fc0fe4-fd68-4340-b90e-e739b4b3534e\") "
Oct 31 13:06:03.436379 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:03.435756    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-xks-ca" pod "36fc0fe4-fd68-4340-b90e-e739b4b3534e" (UID: "36fc0fe4-fd68-4340-b90e-e739b4b3534e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:06:03.436379 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:03.436104    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-openshift-ca" pod "36fc0fe4-fd68-4340-b90e-e739b4b3534e" (UID: "36fc0fe4-fd68-4340-b90e-e739b4b3534e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:06:03.438699 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:03.438654    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36fc0fe4-fd68-4340-b90e-e739b4b3534e-kube-api-access-28msj" pod "36fc0fe4-fd68-4340-b90e-e739b4b3534e" (UID: "36fc0fe4-fd68-4340-b90e-e739b4b3534e"). InnerVolumeSpecName "kube-api-access-28msj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:06:03.532996 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:03.532533    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28msj\" (UniqueName: \"kubernetes.io/projected/36fc0fe4-fd68-4340-b90e-e739b4b3534e-kube-api-access-28msj\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:03.532996 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:03.532589    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:03.532996 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:03.532607    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36fc0fe4-fd68-4340-b90e-e739b4b3534e-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:04.009753 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:04.009720    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"
Oct 31 13:06:04.009941 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:04.009752    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2" event={"ID":"36fc0fe4-fd68-4340-b90e-e739b4b3534e","Type":"ContainerDied","Data":"61d12e25be9decaf8b8743f94563d2ff26ea5d89d89dca74fcb403c6b440c6ce"}
Oct 31 13:06:04.009941 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:04.009800    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61d12e25be9decaf8b8743f94563d2ff26ea5d89d89dca74fcb403c6b440c6ce"
Oct 31 13:06:15.482328 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.482289    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"]
Oct 31 13:06:15.482838 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.482815    5663 cpu_manager.go:414] "containerMap: removing container" podUID="36fc0fe4-fd68-4340-b90e-e739b4b3534e" containerName="revoke-keys"
Oct 31 13:06:15.482932 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.482842    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36fc0fe4-fd68-4340-b90e-e739b4b3534e" containerName="revoke-keys"
Oct 31 13:06:15.482994 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.482938    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="36fc0fe4-fd68-4340-b90e-e739b4b3534e" containerName="revoke-keys"
Oct 31 13:06:15.485970 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.485949    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:15.489916 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.489892    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:06:15.493457 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.493034    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:06:15.495508 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.495490    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"]
Oct 31 13:06:15.543866 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.543836    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mss9q\" (UID: \"ac60d550-6074-46dd-aab2-636f100792bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:15.544086 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.543880    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhtmw\" (UniqueName: \"kubernetes.io/projected/ac60d550-6074-46dd-aab2-636f100792bd-kube-api-access-dhtmw\") pod \"maas-api-revoke-sub-3d4156f1-mss9q\" (UID: \"ac60d550-6074-46dd-aab2-636f100792bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:15.544086 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.544001    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mss9q\" (UID: \"ac60d550-6074-46dd-aab2-636f100792bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:15.644853 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.644816    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhtmw\" (UniqueName: \"kubernetes.io/projected/ac60d550-6074-46dd-aab2-636f100792bd-kube-api-access-dhtmw\") pod \"maas-api-revoke-sub-3d4156f1-mss9q\" (UID: \"ac60d550-6074-46dd-aab2-636f100792bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:15.645029 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.644913    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mss9q\" (UID: \"ac60d550-6074-46dd-aab2-636f100792bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:15.645029 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.644980    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mss9q\" (UID: \"ac60d550-6074-46dd-aab2-636f100792bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:15.645695 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.645671    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mss9q\" (UID: \"ac60d550-6074-46dd-aab2-636f100792bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:15.647194 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.647169    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mss9q\" (UID: \"ac60d550-6074-46dd-aab2-636f100792bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:15.657599 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.657576    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhtmw\" (UniqueName: \"kubernetes.io/projected/ac60d550-6074-46dd-aab2-636f100792bd-kube-api-access-dhtmw\") pod \"maas-api-revoke-sub-3d4156f1-mss9q\" (UID: \"ac60d550-6074-46dd-aab2-636f100792bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:15.802214 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:15.802135    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:16.076815 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:16.076782    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"]
Oct 31 13:06:16.079764 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:06:16.079736    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac60d550_6074_46dd_aab2_636f100792bd.slice/crio-39ed4e3ec0d0442c0b9771886f275b79aa362a04042f4307c76e54962865a694 WatchSource:0}: Error finding container 39ed4e3ec0d0442c0b9771886f275b79aa362a04042f4307c76e54962865a694: Status 404 returned error can't find the container with id 39ed4e3ec0d0442c0b9771886f275b79aa362a04042f4307c76e54962865a694
Oct 31 13:06:17.066799 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:17.066765    5663 generic.go:356] "Generic (PLEG): container finished" podID="ac60d550-6074-46dd-aab2-636f100792bd" containerID="2cd488a3b8c13a5de3f9edb22d7da1c5ae12a2cdf1c33cf7e6c7f97fb3fff665" exitCode=0
Oct 31 13:06:17.067193 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:17.066827    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q" event={"ID":"ac60d550-6074-46dd-aab2-636f100792bd","Type":"ContainerDied","Data":"2cd488a3b8c13a5de3f9edb22d7da1c5ae12a2cdf1c33cf7e6c7f97fb3fff665"}
Oct 31 13:06:17.067193 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:17.066867    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q" event={"ID":"ac60d550-6074-46dd-aab2-636f100792bd","Type":"ContainerStarted","Data":"39ed4e3ec0d0442c0b9771886f275b79aa362a04042f4307c76e54962865a694"}
Oct 31 13:06:18.195852 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:18.195822    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:18.270146 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:18.270107    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac60d550-6074-46dd-aab2-636f100792bd-kube-api-access-dhtmw\" (UniqueName: \"kubernetes.io/projected/ac60d550-6074-46dd-aab2-636f100792bd-kube-api-access-dhtmw\") pod \"ac60d550-6074-46dd-aab2-636f100792bd\" (UID: \"ac60d550-6074-46dd-aab2-636f100792bd\") "
Oct 31 13:06:18.270375 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:18.270161    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-openshift-ca\") pod \"ac60d550-6074-46dd-aab2-636f100792bd\" (UID: \"ac60d550-6074-46dd-aab2-636f100792bd\") "
Oct 31 13:06:18.270375 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:18.270203    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-xks-ca\") pod \"ac60d550-6074-46dd-aab2-636f100792bd\" (UID: \"ac60d550-6074-46dd-aab2-636f100792bd\") "
Oct 31 13:06:18.270645 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:18.270619    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-openshift-ca" pod "ac60d550-6074-46dd-aab2-636f100792bd" (UID: "ac60d550-6074-46dd-aab2-636f100792bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:06:18.272446 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:18.272422    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac60d550-6074-46dd-aab2-636f100792bd-kube-api-access-dhtmw" pod "ac60d550-6074-46dd-aab2-636f100792bd" (UID: "ac60d550-6074-46dd-aab2-636f100792bd"). InnerVolumeSpecName "kube-api-access-dhtmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:06:18.272558 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:18.272501    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-xks-ca" pod "ac60d550-6074-46dd-aab2-636f100792bd" (UID: "ac60d550-6074-46dd-aab2-636f100792bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:06:18.371160 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:18.371057    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhtmw\" (UniqueName: \"kubernetes.io/projected/ac60d550-6074-46dd-aab2-636f100792bd-kube-api-access-dhtmw\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:18.371160 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:18.371102    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:18.371160 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:18.371113    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac60d550-6074-46dd-aab2-636f100792bd-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:19.079666 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:19.079629    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"
Oct 31 13:06:19.079834 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:19.079628    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q" event={"ID":"ac60d550-6074-46dd-aab2-636f100792bd","Type":"ContainerDied","Data":"39ed4e3ec0d0442c0b9771886f275b79aa362a04042f4307c76e54962865a694"}
Oct 31 13:06:19.079834 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:19.079809    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39ed4e3ec0d0442c0b9771886f275b79aa362a04042f4307c76e54962865a694"
Oct 31 13:06:22.475257 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:06:22.475212    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:06:28.254546 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:28.254503    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-lsjpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:06:28.255088 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:28.254578    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-lsjpx" podUID="6cacf32b-0e2d-48cc-a0be-c99e26159bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:06:34.060336 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.060295    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"]
Oct 31 13:06:34.060870 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.060832    5663 cpu_manager.go:414] "containerMap: removing container" podUID="ac60d550-6074-46dd-aab2-636f100792bd" containerName="revoke-keys"
Oct 31 13:06:34.060870 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.060853    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac60d550-6074-46dd-aab2-636f100792bd" containerName="revoke-keys"
Oct 31 13:06:34.060993 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.060975    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac60d550-6074-46dd-aab2-636f100792bd" containerName="revoke-keys"
Oct 31 13:06:34.065608 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.065582    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.074802 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.074779    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-aa9688-openshift-default-dockercfg-57hxk\""
Oct 31 13:06:34.080900 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.080395    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"]
Oct 31 13:06:34.246712 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.246668    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-certs\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.246919 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.246724    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4l5\" (UniqueName: \"kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-kube-api-access-vq4l5\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.246919 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.246842    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-podinfo\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.246919 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.246910    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istiod-ca-cert\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.247091 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.246999    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-socket\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.247091 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.247037    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-envoy\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.247166 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.247090    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-credential-socket\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.247166 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.247140    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-data\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.247245 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.247168    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-token\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.348198 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.348089    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-token\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.348493 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.348470    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-certs\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.348642 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.348626    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4l5\" (UniqueName: \"kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-kube-api-access-vq4l5\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.348776 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.348759    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-podinfo\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.348934 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.348919    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istiod-ca-cert\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.349058 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.349045    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-socket\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.349166 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.349153    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-envoy\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.349264 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.349251    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-credential-socket\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.349413 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.349398    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-data\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.350331 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.350306    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-certs\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.353455 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.353428    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-data\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.353575 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.353548    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-credential-socket\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.353706 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.353679    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-socket\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.354348 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.354223    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istiod-ca-cert\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.354549 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.354526    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-podinfo\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.355237 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.355218    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-envoy\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.362253 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.362226    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-token\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.362576 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.362560    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4l5\" (UniqueName: \"kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-kube-api-access-vq4l5\") pod \"e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") " pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.380477 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.380452    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:34.659165 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.659138    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"]
Oct 31 13:06:34.661556 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:06:34.661531    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9479cc3_0a2e_4b04_ba94_e2c3ead6fee8.slice/crio-6cb47409d5170e873e02be8b6ffab281cc4a7b487ac9c90c1a79427628dcf18b WatchSource:0}: Error finding container 6cb47409d5170e873e02be8b6ffab281cc4a7b487ac9c90c1a79427628dcf18b: Status 404 returned error can't find the container with id 6cb47409d5170e873e02be8b6ffab281cc4a7b487ac9c90c1a79427628dcf18b
Oct 31 13:06:34.663586 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.663553    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:06:34.663665 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.663616    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:06:34.663665 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:34.663646    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:06:35.161492 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:35.161456    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" event={"ID":"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8","Type":"ContainerStarted","Data":"d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c"}
Oct 31 13:06:35.161492 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:35.161511    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" event={"ID":"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8","Type":"ContainerStarted","Data":"6cb47409d5170e873e02be8b6ffab281cc4a7b487ac9c90c1a79427628dcf18b"}
Oct 31 13:06:35.185323 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:35.185279    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" podStartSLOduration=1.185265683 podStartE2EDuration="1.185265683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:35.184141312 +0000 UTC m=+2319.924567528" watchObservedRunningTime="2026-10-31 13:06:35.185265683 +0000 UTC m=+2319.925691898"
Oct 31 13:06:35.381634 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:35.381604    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:35.386705 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:35.386682    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:36.166655 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:36.166619    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:36.167673 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:36.167651    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:38.595098 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.595054    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"]
Oct 31 13:06:38.599463 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.599438    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.613657 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.613630    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-aa9688-serving-cert\""
Oct 31 13:06:38.613823 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.613729    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 13:06:38.613823 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.613757    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6jkzx\""
Oct 31 13:06:38.626318 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.626288    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"]
Oct 31 13:06:38.785914 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.785881    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"]
Oct 31 13:06:38.789315 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.789294    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"
Oct 31 13:06:38.793837 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.793806    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgztl\" (UniqueName: \"kubernetes.io/projected/119f170f-fd66-4df7-b23e-6672bd16b374-kube-api-access-pgztl\") pod \"maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.793959 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.793851    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.793959 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.793944    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/119f170f-fd66-4df7-b23e-6672bd16b374-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.794070 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.794039    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-metrics-certs\") pod \"maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.795042 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.795023    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-aa9688\""
Oct 31 13:06:38.804431 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.804397    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"]
Oct 31 13:06:38.813735 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.813701    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ed661194-dockercfg-l95gk\""
Oct 31 13:06:38.816811 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.816786    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"]
Oct 31 13:06:38.820487 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.820467    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"
Oct 31 13:06:38.835811 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.835778    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"]
Oct 31 13:06:38.896969 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.896881    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-metrics-certs\") pod \"maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.897818 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.897786    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lr4\" (UniqueName: \"kubernetes.io/projected/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-kube-api-access-95lr4\") pod \"payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j\" (UID: \"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"
Oct 31 13:06:38.898061 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.898043    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j\" (UID: \"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"
Oct 31 13:06:38.898222 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.898203    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgztl\" (UniqueName: \"kubernetes.io/projected/119f170f-fd66-4df7-b23e-6672bd16b374-kube-api-access-pgztl\") pod \"maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.898384 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.898344    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.898532 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.898517    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/119f170f-fd66-4df7-b23e-6672bd16b374-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.899430 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.899409    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/119f170f-fd66-4df7-b23e-6672bd16b374-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.902150 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.902120    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-metrics-certs\") pod \"maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.902751 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.902725    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.914145 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.913755    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgztl\" (UniqueName: \"kubernetes.io/projected/119f170f-fd66-4df7-b23e-6672bd16b374-kube-api-access-pgztl\") pod \"maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.914145 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.914024    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:38.999105 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.999073    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4d6\" (UniqueName: \"kubernetes.io/projected/05d80969-0276-497c-bcba-e5d0a60e12b9-kube-api-access-nf4d6\") pod \"payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9\" (UID: \"05d80969-0276-497c-bcba-e5d0a60e12b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"
Oct 31 13:06:38.999271 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.999188    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lr4\" (UniqueName: \"kubernetes.io/projected/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-kube-api-access-95lr4\") pod \"payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j\" (UID: \"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"
Oct 31 13:06:38.999271 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.999245    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j\" (UID: \"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"
Oct 31 13:06:38.999346 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:38.999288    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d80969-0276-497c-bcba-e5d0a60e12b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9\" (UID: \"05d80969-0276-497c-bcba-e5d0a60e12b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"
Oct 31 13:06:39.000161 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:39.000100    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j\" (UID: \"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"
Oct 31 13:06:39.012625 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:39.012593    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lr4\" (UniqueName: \"kubernetes.io/projected/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-kube-api-access-95lr4\") pod \"payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j\" (UID: \"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"
Oct 31 13:06:39.100398 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:39.100341    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d80969-0276-497c-bcba-e5d0a60e12b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9\" (UID: \"05d80969-0276-497c-bcba-e5d0a60e12b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"
Oct 31 13:06:39.100398 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:39.100405    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4d6\" (UniqueName: \"kubernetes.io/projected/05d80969-0276-497c-bcba-e5d0a60e12b9-kube-api-access-nf4d6\") pod \"payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9\" (UID: \"05d80969-0276-497c-bcba-e5d0a60e12b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"
Oct 31 13:06:39.101098 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:39.101066    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d80969-0276-497c-bcba-e5d0a60e12b9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9\" (UID: \"05d80969-0276-497c-bcba-e5d0a60e12b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"
Oct 31 13:06:39.103308 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:39.103285    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"
Oct 31 13:06:39.112705 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:39.112679    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4d6\" (UniqueName: \"kubernetes.io/projected/05d80969-0276-497c-bcba-e5d0a60e12b9-kube-api-access-nf4d6\") pod \"payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9\" (UID: \"05d80969-0276-497c-bcba-e5d0a60e12b9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"
Oct 31 13:06:39.135912 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:39.135865    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"
Oct 31 13:06:39.196458 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:39.196336    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"]
Oct 31 13:06:39.200597 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:06:39.200569    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119f170f_fd66_4df7_b23e_6672bd16b374.slice/crio-f58581da88c8121f65d2122ef045ea1e419f738a034c881fc397bcfa9986ebf6 WatchSource:0}: Error finding container f58581da88c8121f65d2122ef045ea1e419f738a034c881fc397bcfa9986ebf6: Status 404 returned error can't find the container with id f58581da88c8121f65d2122ef045ea1e419f738a034c881fc397bcfa9986ebf6
Oct 31 13:06:39.203540 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:39.203309    5663 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:06:39.403100 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:39.403018    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"]
Oct 31 13:06:39.406183 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:06:39.406154    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7ce312_cf3e_4f1b_ac7a_e2365c8707ae.slice/crio-77d8d29cc65ed182b7514488fc0f3ab3bf1d1f83328c362299b63e89c909bb57 WatchSource:0}: Error finding container 77d8d29cc65ed182b7514488fc0f3ab3bf1d1f83328c362299b63e89c909bb57: Status 404 returned error can't find the container with id 77d8d29cc65ed182b7514488fc0f3ab3bf1d1f83328c362299b63e89c909bb57
Oct 31 13:06:39.422131 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:39.422096    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"]
Oct 31 13:06:39.424931 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:06:39.424906    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d80969_0276_497c_bcba_e5d0a60e12b9.slice/crio-b983a4ae112a67a24a1c7529fb9c59a32b13eaae2120c146efe8e1be04b2993d WatchSource:0}: Error finding container b983a4ae112a67a24a1c7529fb9c59a32b13eaae2120c146efe8e1be04b2993d: Status 404 returned error can't find the container with id b983a4ae112a67a24a1c7529fb9c59a32b13eaae2120c146efe8e1be04b2993d
Oct 31 13:06:40.195550 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:40.195510    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j" event={"ID":"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae","Type":"ContainerStarted","Data":"2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2"}
Oct 31 13:06:40.196018 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:40.195575    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"
Oct 31 13:06:40.196018 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:40.195602    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j" event={"ID":"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae","Type":"ContainerStarted","Data":"77d8d29cc65ed182b7514488fc0f3ab3bf1d1f83328c362299b63e89c909bb57"}
Oct 31 13:06:40.197220 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:40.197193    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd" event={"ID":"119f170f-fd66-4df7-b23e-6672bd16b374","Type":"ContainerStarted","Data":"a9bcfecdd2f91ef16954a5b7924889dd26bd2decd55e0a869a4f9a04c53f94b7"}
Oct 31 13:06:40.197368 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:40.197244    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd" event={"ID":"119f170f-fd66-4df7-b23e-6672bd16b374","Type":"ContainerStarted","Data":"f58581da88c8121f65d2122ef045ea1e419f738a034c881fc397bcfa9986ebf6"}
Oct 31 13:06:40.197368 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:40.197273    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:40.199015 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:40.198988    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9" event={"ID":"05d80969-0276-497c-bcba-e5d0a60e12b9","Type":"ContainerStarted","Data":"229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607"}
Oct 31 13:06:40.199096 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:40.199047    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"
Oct 31 13:06:40.199096 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:40.199072    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9" event={"ID":"05d80969-0276-497c-bcba-e5d0a60e12b9","Type":"ContainerStarted","Data":"b983a4ae112a67a24a1c7529fb9c59a32b13eaae2120c146efe8e1be04b2993d"}
Oct 31 13:06:40.213906 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:40.213867    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j" podStartSLOduration=2.213851045 podStartE2EDuration="2.213851045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:40.213187606 +0000 UTC m=+2324.953613822" watchObservedRunningTime="2026-10-31 13:06:40.213851045 +0000 UTC m=+2324.954277260"
Oct 31 13:06:40.232707 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:40.232666    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9" podStartSLOduration=2.232652775 podStartE2EDuration="2.232652775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:40.231513262 +0000 UTC m=+2324.971939481" watchObservedRunningTime="2026-10-31 13:06:40.232652775 +0000 UTC m=+2324.973078992"
Oct 31 13:06:40.252238 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:40.252192    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd" podStartSLOduration=1.748672881 podStartE2EDuration="2.252179355s" totalImagesPullingTime="503.506474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:40.250617241 +0000 UTC m=+2324.991043456" watchObservedRunningTime="2026-10-31 13:06:40.252179355 +0000 UTC m=+2324.992605598"
Oct 31 13:06:46.210576 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:46.210545    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:06:51.168520 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:51.168477    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 13:06:51.168957 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:51.168548    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 13:06:51.207736 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:51.207708    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"
Oct 31 13:06:51.207948 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:51.207780    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"
Oct 31 13:06:51.256580 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:51.256544    5663 generic.go:356] "Generic (PLEG): container finished" podID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerID="d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c" exitCode=0
Oct 31 13:06:51.256580 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:51.256567    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" event={"ID":"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8","Type":"ContainerDied","Data":"d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c"}
Oct 31 13:06:51.256992 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:51.256978    5663 scope.go:122] "RemoveContainer" containerID="d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c"
Oct 31 13:06:51.258237 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:51.258208    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:06:51.258388 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:51.258268    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:06:51.258388 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:51.258301    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:06:52.284779 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:52.284744    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" event={"ID":"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8","Type":"ContainerStarted","Data":"deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57"}
Oct 31 13:06:52.381376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:52.380964    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:52.381376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:52.381116    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 13:06:52.381376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:52.381200    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 13:06:52.498477 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:06:52.498431    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:06:53.386375 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:53.385801    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 13:06:53.386375 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:53.385867    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 13:06:54.381870 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:54.381826    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 13:06:54.382085 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:54.381900    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 13:06:55.182033 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.181992    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"]
Oct 31 13:06:55.189700 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.189664    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.195293 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.195257    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-aa9688\"/\"openshift-service-ca.crt\""
Oct 31 13:06:55.195293 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.195257    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-aa9688\"/\"default-dockercfg-l7zbv\""
Oct 31 13:06:55.195576 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.195512    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-aa9688\"/\"kube-root-ca.crt\""
Oct 31 13:06:55.195937 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.195916    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-aa9688\"/\"simulator-single-main-aa9688-kserve-self-signed-certs\""
Oct 31 13:06:55.201499 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.201063    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"]
Oct 31 13:06:55.311322 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.311200    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-tmp-dir\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.311322 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.311245    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-model-cache\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.311604 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.311341    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3151543b-1c24-40fb-9195-41467e72a967-tls-certs\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.311604 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.311479    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-dshm\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.311604 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.311511    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6c92\" (UniqueName: \"kubernetes.io/projected/3151543b-1c24-40fb-9195-41467e72a967-kube-api-access-c6c92\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.311604 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.311537    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-home\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.388503 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.384831    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 13:06:55.388503 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.384891    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 13:06:55.412339 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.412299    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-tmp-dir\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.412623 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.412413    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-model-cache\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.412623 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.412457    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3151543b-1c24-40fb-9195-41467e72a967-tls-certs\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.412623 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.412507    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-dshm\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.412623 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.412524    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6c92\" (UniqueName: \"kubernetes.io/projected/3151543b-1c24-40fb-9195-41467e72a967-kube-api-access-c6c92\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.412623 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.412541    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-home\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.412920 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.412870    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-model-cache\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.412980 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.412921    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-home\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.413089 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.413071    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-tmp-dir\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.415426 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.415405    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-dshm\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.415858 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.415836    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3151543b-1c24-40fb-9195-41467e72a967-tls-certs\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.425100 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.425071    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6c92\" (UniqueName: \"kubernetes.io/projected/3151543b-1c24-40fb-9195-41467e72a967-kube-api-access-c6c92\") pod \"simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") " pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.518646 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.518560    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:55.827438 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:55.827413    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"]
Oct 31 13:06:55.829987 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:06:55.829954    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3151543b_1c24_40fb_9195_41467e72a967.slice/crio-a50d5c9ccd276f62b928afebb5141fb5b1e6523e11111c02304bbf53ac2292cf WatchSource:0}: Error finding container a50d5c9ccd276f62b928afebb5141fb5b1e6523e11111c02304bbf53ac2292cf: Status 404 returned error can't find the container with id a50d5c9ccd276f62b928afebb5141fb5b1e6523e11111c02304bbf53ac2292cf
Oct 31 13:06:56.312331 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:56.312296    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89" event={"ID":"3151543b-1c24-40fb-9195-41467e72a967","Type":"ContainerStarted","Data":"fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598"}
Oct 31 13:06:56.312845 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:56.312340    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89" event={"ID":"3151543b-1c24-40fb-9195-41467e72a967","Type":"ContainerStarted","Data":"a50d5c9ccd276f62b928afebb5141fb5b1e6523e11111c02304bbf53ac2292cf"}
Oct 31 13:06:56.335207 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:56.335154    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89" podStartSLOduration=1.335139371 podStartE2EDuration="1.335139371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:56.333578545 +0000 UTC m=+2341.074004761" watchObservedRunningTime="2026-10-31 13:06:56.335139371 +0000 UTC m=+2341.075565623"
Oct 31 13:06:56.383827 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:56.383786    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 13:06:56.384078 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:56.383870    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 13:06:56.519645 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:56.519610    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:06:57.715181 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:57.715151    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:57.715659 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:57.715464    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:57.716283 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:57.716259    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:06:59.666037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:59.666003    5663 scope.go:122] "RemoveContainer" containerID="e4ffcfadce8dddd8ac106dbdefacf00190d72c58aa25ec4c3ed665e4ceefd92d"
Oct 31 13:06:59.674581 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:59.674565    5663 scope.go:122] "RemoveContainer" containerID="e04676dac356f53bb192ffc06a34a67522e830c81c00b8776471dffeadcb9d7e"
Oct 31 13:06:59.682252 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:06:59.682221    5663 scope.go:122] "RemoveContainer" containerID="4c95289927ac8d5b667e40028c3fa867f8ed34fd37c10337e125a242616a7336"
Oct 31 13:07:05.519572 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:05.519476    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:07:05.532517 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:05.532485    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:07:06.369499 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:06.369470    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:07:22.475453 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:07:22.475413    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:07:39.903915 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:39.903881    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"]
Oct 31 13:07:45.104640 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:45.104596    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"]
Oct 31 13:07:45.125455 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:45.125419    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"]
Oct 31 13:07:46.092762 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.092724    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"]
Oct 31 13:07:46.093045 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.092948    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd" podUID="119f170f-fd66-4df7-b23e-6672bd16b374" containerName="maas-api" containerID="cri-o://a9bcfecdd2f91ef16954a5b7924889dd26bd2decd55e0a869a4f9a04c53f94b7" gracePeriod=30
Oct 31 13:07:46.206054 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.206012    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd" podUID="119f170f-fd66-4df7-b23e-6672bd16b374" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.165:8443/health\": dial tcp 10.133.0.165:8443: connect: connection refused"
Oct 31 13:07:46.538451 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.538420    5663 generic.go:356] "Generic (PLEG): container finished" podID="119f170f-fd66-4df7-b23e-6672bd16b374" containerID="a9bcfecdd2f91ef16954a5b7924889dd26bd2decd55e0a869a4f9a04c53f94b7" exitCode=0
Oct 31 13:07:46.538574 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.538463    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd" event={"ID":"119f170f-fd66-4df7-b23e-6672bd16b374","Type":"ContainerDied","Data":"a9bcfecdd2f91ef16954a5b7924889dd26bd2decd55e0a869a4f9a04c53f94b7"}
Oct 31 13:07:46.538574 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.538514    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd" event={"ID":"119f170f-fd66-4df7-b23e-6672bd16b374","Type":"ContainerDied","Data":"f58581da88c8121f65d2122ef045ea1e419f738a034c881fc397bcfa9986ebf6"}
Oct 31 13:07:46.538574 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.538535    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f58581da88c8121f65d2122ef045ea1e419f738a034c881fc397bcfa9986ebf6"
Oct 31 13:07:46.543633 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.543616    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:07:46.618239 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.618205    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119f170f-fd66-4df7-b23e-6672bd16b374-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/119f170f-fd66-4df7-b23e-6672bd16b374-openshift-service-ca\") pod \"119f170f-fd66-4df7-b23e-6672bd16b374\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") "
Oct 31 13:07:46.618471 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.618278    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119f170f-fd66-4df7-b23e-6672bd16b374-kube-api-access-pgztl\" (UniqueName: \"kubernetes.io/projected/119f170f-fd66-4df7-b23e-6672bd16b374-kube-api-access-pgztl\") pod \"119f170f-fd66-4df7-b23e-6672bd16b374\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") "
Oct 31 13:07:46.618471 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.618307    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-maas-api-tls\") pod \"119f170f-fd66-4df7-b23e-6672bd16b374\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") "
Oct 31 13:07:46.618471 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.618380    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-metrics-certs\" (UniqueName: \"kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-metrics-certs\") pod \"119f170f-fd66-4df7-b23e-6672bd16b374\" (UID: \"119f170f-fd66-4df7-b23e-6672bd16b374\") "
Oct 31 13:07:46.618653 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.618625    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119f170f-fd66-4df7-b23e-6672bd16b374-openshift-service-ca" pod "119f170f-fd66-4df7-b23e-6672bd16b374" (UID: "119f170f-fd66-4df7-b23e-6672bd16b374"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:07:46.618765 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.618748    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/119f170f-fd66-4df7-b23e-6672bd16b374-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:46.620445 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.620411    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-metrics-certs" pod "119f170f-fd66-4df7-b23e-6672bd16b374" (UID: "119f170f-fd66-4df7-b23e-6672bd16b374"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:07:46.620445 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.620433    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-maas-api-tls" pod "119f170f-fd66-4df7-b23e-6672bd16b374" (UID: "119f170f-fd66-4df7-b23e-6672bd16b374"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:07:46.620640 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.620480    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119f170f-fd66-4df7-b23e-6672bd16b374-kube-api-access-pgztl" pod "119f170f-fd66-4df7-b23e-6672bd16b374" (UID: "119f170f-fd66-4df7-b23e-6672bd16b374"). InnerVolumeSpecName "kube-api-access-pgztl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:07:46.719936 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.719904    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:46.719936 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.719935    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgztl\" (UniqueName: \"kubernetes.io/projected/119f170f-fd66-4df7-b23e-6672bd16b374-kube-api-access-pgztl\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:46.720162 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:46.719948    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/119f170f-fd66-4df7-b23e-6672bd16b374-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:47.542338 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:47.542305    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"
Oct 31 13:07:47.559791 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:47.559760    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"]
Oct 31 13:07:47.565587 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:47.565560    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aa9688-765c65744c-9d9nd"]
Oct 31 13:07:49.437232 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:49.437194    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119f170f-fd66-4df7-b23e-6672bd16b374" path="/var/lib/kubelet/pods/119f170f-fd66-4df7-b23e-6672bd16b374/volumes"
Oct 31 13:07:50.105227 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:50.105162    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9" podUID="05d80969-0276-497c-bcba-e5d0a60e12b9" containerName="payload-processing" containerID="cri-o://229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607" gracePeriod=25
Oct 31 13:07:50.126723 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:50.126686    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j" podUID="4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae" containerName="payload-pre-processing" containerID="cri-o://2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2" gracePeriod=25
Oct 31 13:07:51.206880 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:51.206840    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9" podUID="05d80969-0276-497c-bcba-e5d0a60e12b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:07:51.207322 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:51.206839    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j" podUID="4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:07:52.475289 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:07:52.475256    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:07:52.913431 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:52.913395    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"]
Oct 31 13:07:52.913920 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:52.913883    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy" containerID="cri-o://deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57" gracePeriod=30
Oct 31 13:07:54.078332 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.078295    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"]
Oct 31 13:07:54.083317 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.083293    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mss9q"]
Oct 31 13:07:54.107250 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.107224    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"]
Oct 31 13:07:54.107662 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.107647    5663 cpu_manager.go:414] "containerMap: removing container" podUID="119f170f-fd66-4df7-b23e-6672bd16b374" containerName="maas-api"
Oct 31 13:07:54.107731 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.107663    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119f170f-fd66-4df7-b23e-6672bd16b374" containerName="maas-api"
Oct 31 13:07:54.107731 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.107729    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="119f170f-fd66-4df7-b23e-6672bd16b374" containerName="maas-api"
Oct 31 13:07:54.111039 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.111023    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:54.114655 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.114637    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:07:54.117790 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.117572    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:07:54.120524 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.120504    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"]
Oct 31 13:07:54.187681 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.187648    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-96qkg\" (UID: \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:54.187681 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.187685    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-96qkg\" (UID: \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:54.187867 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.187807    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrknk\" (UniqueName: \"kubernetes.io/projected/b1b4dc9d-761c-4ad1-9087-3e697cf17416-kube-api-access-mrknk\") pod \"maas-api-revoke-sub-3d4156f1-96qkg\" (UID: \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:54.288626 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.288584    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-96qkg\" (UID: \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:54.288787 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.288676    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-96qkg\" (UID: \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:54.288879 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.288800    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrknk\" (UniqueName: \"kubernetes.io/projected/b1b4dc9d-761c-4ad1-9087-3e697cf17416-kube-api-access-mrknk\") pod \"maas-api-revoke-sub-3d4156f1-96qkg\" (UID: \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:54.289394 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.289347    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-96qkg\" (UID: \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:54.290907 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.290882    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-96qkg\" (UID: \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:54.300765 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.300742    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrknk\" (UniqueName: \"kubernetes.io/projected/b1b4dc9d-761c-4ad1-9087-3e697cf17416-kube-api-access-mrknk\") pod \"maas-api-revoke-sub-3d4156f1-96qkg\" (UID: \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:54.424812 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.424731    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:54.700918 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.700839    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"]
Oct 31 13:07:54.703872 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:07:54.703847    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b4dc9d_761c_4ad1_9087_3e697cf17416.slice/crio-41cc964c4d7ad73c1b5bc74b1c64b2fb1363f919214823888b92f52dc7cedbbd WatchSource:0}: Error finding container 41cc964c4d7ad73c1b5bc74b1c64b2fb1363f919214823888b92f52dc7cedbbd: Status 404 returned error can't find the container with id 41cc964c4d7ad73c1b5bc74b1c64b2fb1363f919214823888b92f52dc7cedbbd
Oct 31 13:07:54.923643 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:54.923594    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89" podUID="3151543b-1c24-40fb-9195-41467e72a967" containerName="main" containerID="cri-o://fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598" gracePeriod=45
Oct 31 13:07:55.161009 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.160984    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:07:55.197568 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.197521    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-tmp-dir\") pod \"3151543b-1c24-40fb-9195-41467e72a967\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") "
Oct 31 13:07:55.197763 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.197619    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-model-cache\") pod \"3151543b-1c24-40fb-9195-41467e72a967\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") "
Oct 31 13:07:55.197763 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.197731    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-dshm\") pod \"3151543b-1c24-40fb-9195-41467e72a967\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") "
Oct 31 13:07:55.197941 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.197802    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3151543b-1c24-40fb-9195-41467e72a967-tls-certs\" (UniqueName: \"kubernetes.io/secret/3151543b-1c24-40fb-9195-41467e72a967-tls-certs\") pod \"3151543b-1c24-40fb-9195-41467e72a967\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") "
Oct 31 13:07:55.197941 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.197883    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3151543b-1c24-40fb-9195-41467e72a967-kube-api-access-c6c92\" (UniqueName: \"kubernetes.io/projected/3151543b-1c24-40fb-9195-41467e72a967-kube-api-access-c6c92\") pod \"3151543b-1c24-40fb-9195-41467e72a967\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") "
Oct 31 13:07:55.198058 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.197944    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-home\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-home\") pod \"3151543b-1c24-40fb-9195-41467e72a967\" (UID: \"3151543b-1c24-40fb-9195-41467e72a967\") "
Oct 31 13:07:55.198058 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.197979    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-tmp-dir" pod "3151543b-1c24-40fb-9195-41467e72a967" (UID: "3151543b-1c24-40fb-9195-41467e72a967"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:07:55.198058 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.198036    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-model-cache" pod "3151543b-1c24-40fb-9195-41467e72a967" (UID: "3151543b-1c24-40fb-9195-41467e72a967"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:07:55.198252 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.198230    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-home" pod "3151543b-1c24-40fb-9195-41467e72a967" (UID: "3151543b-1c24-40fb-9195-41467e72a967"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:07:55.198319 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.198301    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:55.198387 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.198327    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:55.200703 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.200674    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-dshm" pod "3151543b-1c24-40fb-9195-41467e72a967" (UID: "3151543b-1c24-40fb-9195-41467e72a967"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:07:55.200703 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.200693    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3151543b-1c24-40fb-9195-41467e72a967-tls-certs" pod "3151543b-1c24-40fb-9195-41467e72a967" (UID: "3151543b-1c24-40fb-9195-41467e72a967"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:07:55.201189 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.201166    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3151543b-1c24-40fb-9195-41467e72a967-kube-api-access-c6c92" pod "3151543b-1c24-40fb-9195-41467e72a967" (UID: "3151543b-1c24-40fb-9195-41467e72a967"). InnerVolumeSpecName "kube-api-access-c6c92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:07:55.299593 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.299561    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6c92\" (UniqueName: \"kubernetes.io/projected/3151543b-1c24-40fb-9195-41467e72a967-kube-api-access-c6c92\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:55.299593 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.299596    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:55.299884 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.299610    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3151543b-1c24-40fb-9195-41467e72a967-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:55.299884 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.299622    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3151543b-1c24-40fb-9195-41467e72a967-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:55.436818 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.436788    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac60d550-6074-46dd-aab2-636f100792bd" path="/var/lib/kubelet/pods/ac60d550-6074-46dd-aab2-636f100792bd/volumes"
Oct 31 13:07:55.579963 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.579935    5663 generic.go:356] "Generic (PLEG): container finished" podID="b1b4dc9d-761c-4ad1-9087-3e697cf17416" containerID="3c9ddbb63a76b463d85da568903dc4b45f9592c21556be07f076337104613f6e" exitCode=0
Oct 31 13:07:55.580149 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.580028    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg" event={"ID":"b1b4dc9d-761c-4ad1-9087-3e697cf17416","Type":"ContainerDied","Data":"3c9ddbb63a76b463d85da568903dc4b45f9592c21556be07f076337104613f6e"}
Oct 31 13:07:55.580149 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.580097    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg" event={"ID":"b1b4dc9d-761c-4ad1-9087-3e697cf17416","Type":"ContainerStarted","Data":"41cc964c4d7ad73c1b5bc74b1c64b2fb1363f919214823888b92f52dc7cedbbd"}
Oct 31 13:07:55.581296 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.581277    5663 generic.go:356] "Generic (PLEG): container finished" podID="3151543b-1c24-40fb-9195-41467e72a967" containerID="fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598" exitCode=0
Oct 31 13:07:55.581405 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.581323    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"
Oct 31 13:07:55.581405 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.581339    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89" event={"ID":"3151543b-1c24-40fb-9195-41467e72a967","Type":"ContainerDied","Data":"fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598"}
Oct 31 13:07:55.581405 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.581388    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89" event={"ID":"3151543b-1c24-40fb-9195-41467e72a967","Type":"ContainerDied","Data":"a50d5c9ccd276f62b928afebb5141fb5b1e6523e11111c02304bbf53ac2292cf"}
Oct 31 13:07:55.581520 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.581406    5663 scope.go:122] "RemoveContainer" containerID="fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598"
Oct 31 13:07:55.590157 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.590142    5663 scope.go:122] "RemoveContainer" containerID="fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598"
Oct 31 13:07:55.590450 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:07:55.590425    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598\": container with ID starting with fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598 not found: ID does not exist" containerID="fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598"
Oct 31 13:07:55.590564 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.590457    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598"} err="failed to get container status \"fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598\": rpc error: code = NotFound desc = could not find container \"fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598\": container with ID starting with fba85c7d51953cfe4cd9a797e2680d318b7d2db58676f457eef6945525780598 not found: ID does not exist"
Oct 31 13:07:55.612317 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.612291    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"]
Oct 31 13:07:55.617079 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:55.617056    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-aa9688/simulator-single-main-aa9688-kserve-8bd9f9b95-qwn89"]
Oct 31 13:07:56.718963 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:56.718938    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:56.813037 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:56.813004    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b4dc9d-761c-4ad1-9087-3e697cf17416-kube-api-access-mrknk\" (UniqueName: \"kubernetes.io/projected/b1b4dc9d-761c-4ad1-9087-3e697cf17416-kube-api-access-mrknk\") pod \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\" (UID: \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\") "
Oct 31 13:07:56.813200 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:56.813056    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-openshift-ca\") pod \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\" (UID: \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\") "
Oct 31 13:07:56.813200 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:56.813078    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-xks-ca\") pod \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\" (UID: \"b1b4dc9d-761c-4ad1-9087-3e697cf17416\") "
Oct 31 13:07:56.813486 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:56.813449    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-openshift-ca" pod "b1b4dc9d-761c-4ad1-9087-3e697cf17416" (UID: "b1b4dc9d-761c-4ad1-9087-3e697cf17416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:07:56.815164 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:56.815142    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-xks-ca" pod "b1b4dc9d-761c-4ad1-9087-3e697cf17416" (UID: "b1b4dc9d-761c-4ad1-9087-3e697cf17416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:07:56.815300 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:56.815281    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b4dc9d-761c-4ad1-9087-3e697cf17416-kube-api-access-mrknk" pod "b1b4dc9d-761c-4ad1-9087-3e697cf17416" (UID: "b1b4dc9d-761c-4ad1-9087-3e697cf17416"). InnerVolumeSpecName "kube-api-access-mrknk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:07:56.914771 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:56.914695    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrknk\" (UniqueName: \"kubernetes.io/projected/b1b4dc9d-761c-4ad1-9087-3e697cf17416-kube-api-access-mrknk\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:56.914771 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:56.914721    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:56.914771 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:56.914731    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4dc9d-761c-4ad1-9087-3e697cf17416-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:57.436817 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:57.436783    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3151543b-1c24-40fb-9195-41467e72a967" path="/var/lib/kubelet/pods/3151543b-1c24-40fb-9195-41467e72a967/volumes"
Oct 31 13:07:57.592932 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:57.592905    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"
Oct 31 13:07:57.593104 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:57.592910    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg" event={"ID":"b1b4dc9d-761c-4ad1-9087-3e697cf17416","Type":"ContainerDied","Data":"41cc964c4d7ad73c1b5bc74b1c64b2fb1363f919214823888b92f52dc7cedbbd"}
Oct 31 13:07:57.593104 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:57.593023    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41cc964c4d7ad73c1b5bc74b1c64b2fb1363f919214823888b92f52dc7cedbbd"
Oct 31 13:07:57.718075 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:57.717990    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:07:57.718075 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:57.718048    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:07:58.172635 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.172612    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:07:58.227392 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.227362    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istiod-ca-cert\") pod \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") "
Oct 31 13:07:58.227565 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.227410    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-podinfo\") pod \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") "
Oct 31 13:07:58.227565 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.227435    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-credential-socket\") pod \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") "
Oct 31 13:07:58.227565 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.227470    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-data\") pod \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") "
Oct 31 13:07:58.227565 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.227491    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-envoy\") pod \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") "
Oct 31 13:07:58.227565 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.227562    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-certs\") pod \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") "
Oct 31 13:07:58.227830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.227624    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-token\" (UniqueName: \"kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-token\") pod \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") "
Oct 31 13:07:58.227830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.227653    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-kube-api-access-vq4l5\" (UniqueName: \"kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-kube-api-access-vq4l5\") pod \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") "
Oct 31 13:07:58.227830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.227720    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-socket\") pod \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\" (UID: \"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8\") "
Oct 31 13:07:58.227830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.227757    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istiod-ca-cert" pod "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" (UID: "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:07:58.227830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.227822    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-credential-socket" pod "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" (UID: "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:07:58.228175 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.228009    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-socket" pod "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" (UID: "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:07:58.228175 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.228072    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:58.228175 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.228090    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:58.228175 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.228106    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:58.228407 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.228220    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-certs" pod "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" (UID: "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:07:58.228407 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.228280    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-data" pod "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" (UID: "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:07:58.229959 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.229931    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-kube-api-access-vq4l5" pod "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" (UID: "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8"). InnerVolumeSpecName "kube-api-access-vq4l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:07:58.230455 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.230428    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-envoy" pod "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" (UID: "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:07:58.230564 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.230474    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-token" pod "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" (UID: "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:07:58.230564 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.230490    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-podinfo" pod "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" (UID: "a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:07:58.329255 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.329226    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:58.329255 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.329254    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:58.329467 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.329263    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:58.329467 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.329276    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:58.329467 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.329288    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:58.329467 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.329298    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq4l5\" (UniqueName: \"kubernetes.io/projected/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8-kube-api-access-vq4l5\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:58.598474 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.598373    5663 generic.go:356] "Generic (PLEG): container finished" podID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerID="deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57" exitCode=0
Oct 31 13:07:58.598474 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.598430    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"
Oct 31 13:07:58.598474 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.598459    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" event={"ID":"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8","Type":"ContainerDied","Data":"deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57"}
Oct 31 13:07:58.598761 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.598519    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl" event={"ID":"a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8","Type":"ContainerDied","Data":"6cb47409d5170e873e02be8b6ffab281cc4a7b487ac9c90c1a79427628dcf18b"}
Oct 31 13:07:58.598761 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.598540    5663 scope.go:122] "RemoveContainer" containerID="deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57"
Oct 31 13:07:58.615348 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.615327    5663 scope.go:122] "RemoveContainer" containerID="d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c"
Oct 31 13:07:58.623294 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.623274    5663 scope.go:122] "RemoveContainer" containerID="deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57"
Oct 31 13:07:58.623555 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:07:58.623536    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57\": container with ID starting with deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57 not found: ID does not exist" containerID="deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57"
Oct 31 13:07:58.623621 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.623563    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57"} err="failed to get container status \"deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57\": rpc error: code = NotFound desc = could not find container \"deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57\": container with ID starting with deb63f18619274eb7d67a0a4d048fd4c27dfe0ea184b8eef46aa31d43b412b57 not found: ID does not exist"
Oct 31 13:07:58.623621 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.623578    5663 scope.go:122] "RemoveContainer" containerID="d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c"
Oct 31 13:07:58.623781 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:07:58.623764    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c\": container with ID starting with d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c not found: ID does not exist" containerID="d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c"
Oct 31 13:07:58.623823 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.623785    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c"} err="failed to get container status \"d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c\": rpc error: code = NotFound desc = could not find container \"d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c\": container with ID starting with d954e4888589e9cac881180cb13178e98cded8eb64b20e6687c45712dc8fff6c not found: ID does not exist"
Oct 31 13:07:58.627966 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.627938    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"]
Oct 31 13:07:58.632906 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:58.632882    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aa9688-openshift-default-86ffccbdf-9rxnl"]
Oct 31 13:07:59.437140 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:07:59.437097    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" path="/var/lib/kubelet/pods/a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8/volumes"
Oct 31 13:08:01.206423 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:01.206386    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j" podUID="4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:08:01.206848 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:01.206398    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9" podUID="05d80969-0276-497c-bcba-e5d0a60e12b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:08:10.366288 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.366222    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"
Oct 31 13:08:10.392975 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.392947    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"
Oct 31 13:08:10.457578 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.457548    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d80969-0276-497c-bcba-e5d0a60e12b9-kube-api-access-nf4d6\" (UniqueName: \"kubernetes.io/projected/05d80969-0276-497c-bcba-e5d0a60e12b9-kube-api-access-nf4d6\") pod \"05d80969-0276-497c-bcba-e5d0a60e12b9\" (UID: \"05d80969-0276-497c-bcba-e5d0a60e12b9\") "
Oct 31 13:08:10.457770 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.457627    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d80969-0276-497c-bcba-e5d0a60e12b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d80969-0276-497c-bcba-e5d0a60e12b9-plugins-config-volume\") pod \"05d80969-0276-497c-bcba-e5d0a60e12b9\" (UID: \"05d80969-0276-497c-bcba-e5d0a60e12b9\") "
Oct 31 13:08:10.458028 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.458003    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d80969-0276-497c-bcba-e5d0a60e12b9-plugins-config-volume" pod "05d80969-0276-497c-bcba-e5d0a60e12b9" (UID: "05d80969-0276-497c-bcba-e5d0a60e12b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:08:10.459639 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.459618    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d80969-0276-497c-bcba-e5d0a60e12b9-kube-api-access-nf4d6" pod "05d80969-0276-497c-bcba-e5d0a60e12b9" (UID: "05d80969-0276-497c-bcba-e5d0a60e12b9"). InnerVolumeSpecName "kube-api-access-nf4d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:08:10.558816 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.558784    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-kube-api-access-95lr4\" (UniqueName: \"kubernetes.io/projected/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-kube-api-access-95lr4\") pod \"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae\" (UID: \"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae\") "
Oct 31 13:08:10.558992 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.558875    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-plugins-config-volume\") pod \"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae\" (UID: \"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae\") "
Oct 31 13:08:10.559119 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.559103    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf4d6\" (UniqueName: \"kubernetes.io/projected/05d80969-0276-497c-bcba-e5d0a60e12b9-kube-api-access-nf4d6\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:08:10.559188 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.559123    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d80969-0276-497c-bcba-e5d0a60e12b9-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:08:10.559294 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.559269    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-plugins-config-volume" pod "4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae" (UID: "4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:08:10.560964 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.560943    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-kube-api-access-95lr4" pod "4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae" (UID: "4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae"). InnerVolumeSpecName "kube-api-access-95lr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:08:10.655728 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.655654    5663 generic.go:356] "Generic (PLEG): container finished" podID="05d80969-0276-497c-bcba-e5d0a60e12b9" containerID="229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607" exitCode=0
Oct 31 13:08:10.655728 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.655698    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"
Oct 31 13:08:10.655950 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.655741    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9" event={"ID":"05d80969-0276-497c-bcba-e5d0a60e12b9","Type":"ContainerDied","Data":"229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607"}
Oct 31 13:08:10.655950 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.655802    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9" event={"ID":"05d80969-0276-497c-bcba-e5d0a60e12b9","Type":"ContainerDied","Data":"b983a4ae112a67a24a1c7529fb9c59a32b13eaae2120c146efe8e1be04b2993d"}
Oct 31 13:08:10.655950 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.655834    5663 scope.go:122] "RemoveContainer" containerID="229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607"
Oct 31 13:08:10.657007 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.656983    5663 generic.go:356] "Generic (PLEG): container finished" podID="4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae" containerID="2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2" exitCode=0
Oct 31 13:08:10.657100 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.657010    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j" event={"ID":"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae","Type":"ContainerDied","Data":"2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2"}
Oct 31 13:08:10.657100 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.657033    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"
Oct 31 13:08:10.657100 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.657073    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j" event={"ID":"4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae","Type":"ContainerDied","Data":"77d8d29cc65ed182b7514488fc0f3ab3bf1d1f83328c362299b63e89c909bb57"}
Oct 31 13:08:10.659608 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.659583    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:08:10.659704 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.659613    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95lr4\" (UniqueName: \"kubernetes.io/projected/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae-kube-api-access-95lr4\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:08:10.665508 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.665487    5663 scope.go:122] "RemoveContainer" containerID="229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607"
Oct 31 13:08:10.665768 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:08:10.665747    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607\": container with ID starting with 229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607 not found: ID does not exist" containerID="229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607"
Oct 31 13:08:10.665823 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.665776    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607"} err="failed to get container status \"229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607\": rpc error: code = NotFound desc = could not find container \"229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607\": container with ID starting with 229fd2a82aa109237fee38e22267fc3851babea07a0596c4f2df64b0a3350607 not found: ID does not exist"
Oct 31 13:08:10.665823 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.665795    5663 scope.go:122] "RemoveContainer" containerID="2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2"
Oct 31 13:08:10.673368 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.673341    5663 scope.go:122] "RemoveContainer" containerID="2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2"
Oct 31 13:08:10.673654 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:08:10.673636    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2\": container with ID starting with 2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2 not found: ID does not exist" containerID="2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2"
Oct 31 13:08:10.673703 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.673661    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2"} err="failed to get container status \"2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2\": rpc error: code = NotFound desc = could not find container \"2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2\": container with ID starting with 2bb2513b135eddaef62ca16fc27d3e2ca815ab1e7b785f154082a0d401f896c2 not found: ID does not exist"
Oct 31 13:08:10.679741 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.679719    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"]
Oct 31 13:08:10.684347 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.684326    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aa9688-5f5d44fdfv79w9"]
Oct 31 13:08:10.695683 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.695663    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"]
Oct 31 13:08:10.699930 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:10.699910    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa9688-55846xwg2j"]
Oct 31 13:08:11.444242 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:11.444154    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d80969-0276-497c-bcba-e5d0a60e12b9" path="/var/lib/kubelet/pods/05d80969-0276-497c-bcba-e5d0a60e12b9/volumes"
Oct 31 13:08:11.444692 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:08:11.444673    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae" path="/var/lib/kubelet/pods/4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae/volumes"
Oct 31 13:08:22.475057 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:08:22.475022    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:08:52.475234 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:08:52.475153    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:09:22.474704 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:09:22.474666    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:09:40.237661 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.237622    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"]
Oct 31 13:09:40.238177 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238148    5663 cpu_manager.go:414] "containerMap: removing container" podUID="05d80969-0276-497c-bcba-e5d0a60e12b9" containerName="payload-processing"
Oct 31 13:09:40.238177 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238166    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d80969-0276-497c-bcba-e5d0a60e12b9" containerName="payload-processing"
Oct 31 13:09:40.238303 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238184    5663 cpu_manager.go:414] "containerMap: removing container" podUID="3151543b-1c24-40fb-9195-41467e72a967" containerName="main"
Oct 31 13:09:40.238303 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238192    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3151543b-1c24-40fb-9195-41467e72a967" containerName="main"
Oct 31 13:09:40.238303 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238212    5663 cpu_manager.go:414] "containerMap: removing container" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy"
Oct 31 13:09:40.238303 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238222    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy"
Oct 31 13:09:40.238303 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238245    5663 cpu_manager.go:414] "containerMap: removing container" podUID="4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae" containerName="payload-pre-processing"
Oct 31 13:09:40.238303 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238252    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae" containerName="payload-pre-processing"
Oct 31 13:09:40.238303 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238266    5663 cpu_manager.go:414] "containerMap: removing container" podUID="b1b4dc9d-761c-4ad1-9087-3e697cf17416" containerName="revoke-keys"
Oct 31 13:09:40.238303 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238274    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b4dc9d-761c-4ad1-9087-3e697cf17416" containerName="revoke-keys"
Oct 31 13:09:40.238748 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238381    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d80969-0276-497c-bcba-e5d0a60e12b9" containerName="payload-processing"
Oct 31 13:09:40.238748 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238393    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="3151543b-1c24-40fb-9195-41467e72a967" containerName="main"
Oct 31 13:09:40.238748 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238405    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy"
Oct 31 13:09:40.238748 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238416    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7ce312-cf3e-4f1b-ac7a-e2365c8707ae" containerName="payload-pre-processing"
Oct 31 13:09:40.238748 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238426    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b4dc9d-761c-4ad1-9087-3e697cf17416" containerName="revoke-keys"
Oct 31 13:09:40.238748 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.238436    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy"
Oct 31 13:09:40.241563 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.241544    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:40.245926 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.245906    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:09:40.246028 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.245998    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:09:40.252320 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.251556    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"]
Oct 31 13:09:40.405607 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.405564    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rp6g4\" (UID: \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:40.405784 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.405638    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcrsb\" (UniqueName: \"kubernetes.io/projected/9fcbaba0-4c84-4304-8b14-92d668b84b3e-kube-api-access-xcrsb\") pod \"maas-api-revoke-sub-d576e68b-rp6g4\" (UID: \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:40.405784 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.405731    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rp6g4\" (UID: \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:40.507145 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.507039    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rp6g4\" (UID: \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:40.507145 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.507109    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rp6g4\" (UID: \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:40.507420 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.507155    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcrsb\" (UniqueName: \"kubernetes.io/projected/9fcbaba0-4c84-4304-8b14-92d668b84b3e-kube-api-access-xcrsb\") pod \"maas-api-revoke-sub-d576e68b-rp6g4\" (UID: \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:40.507751 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.507728    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rp6g4\" (UID: \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:40.509485 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.509462    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rp6g4\" (UID: \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:40.519052 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.519029    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcrsb\" (UniqueName: \"kubernetes.io/projected/9fcbaba0-4c84-4304-8b14-92d668b84b3e-kube-api-access-xcrsb\") pod \"maas-api-revoke-sub-d576e68b-rp6g4\" (UID: \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:40.554648 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.554622    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:40.825753 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:40.825724    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"]
Oct 31 13:09:40.828394 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:09:40.828340    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcbaba0_4c84_4304_8b14_92d668b84b3e.slice/crio-c09bd26e42f127ba45ded7573f106af78e948b010d056b6ef661aa390ccbf60e WatchSource:0}: Error finding container c09bd26e42f127ba45ded7573f106af78e948b010d056b6ef661aa390ccbf60e: Status 404 returned error can't find the container with id c09bd26e42f127ba45ded7573f106af78e948b010d056b6ef661aa390ccbf60e
Oct 31 13:09:41.046393 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:41.046337    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4" event={"ID":"9fcbaba0-4c84-4304-8b14-92d668b84b3e","Type":"ContainerStarted","Data":"c09bd26e42f127ba45ded7573f106af78e948b010d056b6ef661aa390ccbf60e"}
Oct 31 13:09:42.051518 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:42.051427    5663 generic.go:356] "Generic (PLEG): container finished" podID="9fcbaba0-4c84-4304-8b14-92d668b84b3e" containerID="b15807c018aac104629af218566c5a8f59ad0058a69ad754c33a0ad4d02537a2" exitCode=0
Oct 31 13:09:42.051959 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:42.051514    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4" event={"ID":"9fcbaba0-4c84-4304-8b14-92d668b84b3e","Type":"ContainerDied","Data":"b15807c018aac104629af218566c5a8f59ad0058a69ad754c33a0ad4d02537a2"}
Oct 31 13:09:43.182406 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:43.182348    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:43.331641 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:43.331561    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-xks-ca\") pod \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\" (UID: \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\") "
Oct 31 13:09:43.331641 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:43.331611    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-openshift-ca\") pod \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\" (UID: \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\") "
Oct 31 13:09:43.331834 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:43.331696    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fcbaba0-4c84-4304-8b14-92d668b84b3e-kube-api-access-xcrsb\" (UniqueName: \"kubernetes.io/projected/9fcbaba0-4c84-4304-8b14-92d668b84b3e-kube-api-access-xcrsb\") pod \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\" (UID: \"9fcbaba0-4c84-4304-8b14-92d668b84b3e\") "
Oct 31 13:09:43.331994 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:43.331972    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-openshift-ca" pod "9fcbaba0-4c84-4304-8b14-92d668b84b3e" (UID: "9fcbaba0-4c84-4304-8b14-92d668b84b3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:09:43.333754 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:43.333732    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-xks-ca" pod "9fcbaba0-4c84-4304-8b14-92d668b84b3e" (UID: "9fcbaba0-4c84-4304-8b14-92d668b84b3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:09:43.333754 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:43.333748    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcbaba0-4c84-4304-8b14-92d668b84b3e-kube-api-access-xcrsb" pod "9fcbaba0-4c84-4304-8b14-92d668b84b3e" (UID: "9fcbaba0-4c84-4304-8b14-92d668b84b3e"). InnerVolumeSpecName "kube-api-access-xcrsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:09:43.432477 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:43.432442    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcrsb\" (UniqueName: \"kubernetes.io/projected/9fcbaba0-4c84-4304-8b14-92d668b84b3e-kube-api-access-xcrsb\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:43.432477 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:43.432471    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:43.432477 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:43.432482    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fcbaba0-4c84-4304-8b14-92d668b84b3e-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:09:44.060138 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:44.060102    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4" event={"ID":"9fcbaba0-4c84-4304-8b14-92d668b84b3e","Type":"ContainerDied","Data":"c09bd26e42f127ba45ded7573f106af78e948b010d056b6ef661aa390ccbf60e"}
Oct 31 13:09:44.060389 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:44.060156    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c09bd26e42f127ba45ded7573f106af78e948b010d056b6ef661aa390ccbf60e"
Oct 31 13:09:44.060389 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:44.060157    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"
Oct 31 13:09:52.478173 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:09:52.478127    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:09:59.063574 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.063538    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"]
Oct 31 13:09:59.064060 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.063893    5663 cpu_manager.go:414] "containerMap: removing container" podUID="9fcbaba0-4c84-4304-8b14-92d668b84b3e" containerName="revoke-keys"
Oct 31 13:09:59.064060 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.063905    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcbaba0-4c84-4304-8b14-92d668b84b3e" containerName="revoke-keys"
Oct 31 13:09:59.064060 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.063917    5663 cpu_manager.go:414] "containerMap: removing container" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy"
Oct 31 13:09:59.064060 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.063922    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9479cc3-0a2e-4b04-ba94-e2c3ead6fee8" containerName="istio-proxy"
Oct 31 13:09:59.064060 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.063983    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcbaba0-4c84-4304-8b14-92d668b84b3e" containerName="revoke-keys"
Oct 31 13:09:59.067087 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.067069    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:09:59.075947 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.075923    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"]
Oct 31 13:09:59.080303 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.080283    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cz9v2\" (UID: \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:09:59.080456 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.080336    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cz9v2\" (UID: \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:09:59.080519 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.080456    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl2vr\" (UniqueName: \"kubernetes.io/projected/8b2ac3df-7d7f-403d-a723-83a1eca1969d-kube-api-access-wl2vr\") pod \"maas-api-revoke-sub-d2bc93bc-cz9v2\" (UID: \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:09:59.147929 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.147899    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:09:59.148110 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.148018    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:09:59.181734 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.181702    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cz9v2\" (UID: \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:09:59.181887 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.181763    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cz9v2\" (UID: \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:09:59.181887 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.181799    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl2vr\" (UniqueName: \"kubernetes.io/projected/8b2ac3df-7d7f-403d-a723-83a1eca1969d-kube-api-access-wl2vr\") pod \"maas-api-revoke-sub-d2bc93bc-cz9v2\" (UID: \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:09:59.182341 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.182323    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cz9v2\" (UID: \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:09:59.184045 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.184027    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-cz9v2\" (UID: \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:09:59.193773 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.193746    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl2vr\" (UniqueName: \"kubernetes.io/projected/8b2ac3df-7d7f-403d-a723-83a1eca1969d-kube-api-access-wl2vr\") pod \"maas-api-revoke-sub-d2bc93bc-cz9v2\" (UID: \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:09:59.381027 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.380935    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:09:59.653101 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:09:59.652991    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"]
Oct 31 13:09:59.657239 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:09:59.657209    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2ac3df_7d7f_403d_a723_83a1eca1969d.slice/crio-b5b3cf4506ca4ea9973407e9e68eef7e89cc9306c591741fb9c3df8a4a99ec2f WatchSource:0}: Error finding container b5b3cf4506ca4ea9973407e9e68eef7e89cc9306c591741fb9c3df8a4a99ec2f: Status 404 returned error can't find the container with id b5b3cf4506ca4ea9973407e9e68eef7e89cc9306c591741fb9c3df8a4a99ec2f
Oct 31 13:10:00.167052 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:00.167019    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2" event={"ID":"8b2ac3df-7d7f-403d-a723-83a1eca1969d","Type":"ContainerStarted","Data":"4ca37c9735ae8540201c71bd759405981fb7d9e53c1445584f08ad7908718aef"}
Oct 31 13:10:00.167490 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:00.167075    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2" event={"ID":"8b2ac3df-7d7f-403d-a723-83a1eca1969d","Type":"ContainerStarted","Data":"b5b3cf4506ca4ea9973407e9e68eef7e89cc9306c591741fb9c3df8a4a99ec2f"}
Oct 31 13:10:00.185877 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:00.185818    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2" podStartSLOduration=1.185800314 podStartE2EDuration="1.185800314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:00.185138626 +0000 UTC m=+2524.925564846" watchObservedRunningTime="2026-10-31 13:10:00.185800314 +0000 UTC m=+2524.926226531"
Oct 31 13:10:01.172288 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:01.172248    5663 generic.go:356] "Generic (PLEG): container finished" podID="8b2ac3df-7d7f-403d-a723-83a1eca1969d" containerID="4ca37c9735ae8540201c71bd759405981fb7d9e53c1445584f08ad7908718aef" exitCode=0
Oct 31 13:10:01.172716 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:01.172334    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2" event={"ID":"8b2ac3df-7d7f-403d-a723-83a1eca1969d","Type":"ContainerDied","Data":"4ca37c9735ae8540201c71bd759405981fb7d9e53c1445584f08ad7908718aef"}
Oct 31 13:10:02.298779 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:02.298754    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:10:02.407830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:02.407677    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-openshift-ca\") pod \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\" (UID: \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\") "
Oct 31 13:10:02.407830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:02.407778    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b2ac3df-7d7f-403d-a723-83a1eca1969d-kube-api-access-wl2vr\" (UniqueName: \"kubernetes.io/projected/8b2ac3df-7d7f-403d-a723-83a1eca1969d-kube-api-access-wl2vr\") pod \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\" (UID: \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\") "
Oct 31 13:10:02.407830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:02.407801    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-xks-ca\") pod \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\" (UID: \"8b2ac3df-7d7f-403d-a723-83a1eca1969d\") "
Oct 31 13:10:02.408133 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:02.408106    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-openshift-ca" pod "8b2ac3df-7d7f-403d-a723-83a1eca1969d" (UID: "8b2ac3df-7d7f-403d-a723-83a1eca1969d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:10:02.409978 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:02.409953    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2ac3df-7d7f-403d-a723-83a1eca1969d-kube-api-access-wl2vr" pod "8b2ac3df-7d7f-403d-a723-83a1eca1969d" (UID: "8b2ac3df-7d7f-403d-a723-83a1eca1969d"). InnerVolumeSpecName "kube-api-access-wl2vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:10:02.410081 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:02.409980    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-xks-ca" pod "8b2ac3df-7d7f-403d-a723-83a1eca1969d" (UID: "8b2ac3df-7d7f-403d-a723-83a1eca1969d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:10:02.508601 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:02.508513    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:10:02.508601 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:02.508546    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl2vr\" (UniqueName: \"kubernetes.io/projected/8b2ac3df-7d7f-403d-a723-83a1eca1969d-kube-api-access-wl2vr\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:10:02.508601 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:02.508557    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b2ac3df-7d7f-403d-a723-83a1eca1969d-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:10:03.181577 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:03.181547    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"
Oct 31 13:10:03.181577 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:03.181564    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2" event={"ID":"8b2ac3df-7d7f-403d-a723-83a1eca1969d","Type":"ContainerDied","Data":"b5b3cf4506ca4ea9973407e9e68eef7e89cc9306c591741fb9c3df8a4a99ec2f"}
Oct 31 13:10:03.181851 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:03.181618    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5b3cf4506ca4ea9973407e9e68eef7e89cc9306c591741fb9c3df8a4a99ec2f"
Oct 31 13:10:10.975708 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:10.975666    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"]
Oct 31 13:10:10.976295 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:10.976272    5663 cpu_manager.go:414] "containerMap: removing container" podUID="8b2ac3df-7d7f-403d-a723-83a1eca1969d" containerName="revoke-keys"
Oct 31 13:10:10.976383 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:10.976299    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b2ac3df-7d7f-403d-a723-83a1eca1969d" containerName="revoke-keys"
Oct 31 13:10:10.976434 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:10.976410    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b2ac3df-7d7f-403d-a723-83a1eca1969d" containerName="revoke-keys"
Oct 31 13:10:10.979773 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:10.979755    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:10.984744 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:10.984720    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-169e4e-openshift-default-dockercfg-nfrlh\""
Oct 31 13:10:10.991568 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:10.991540    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"]
Oct 31 13:10:11.083540 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.083503    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-socket\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.083717 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.083553    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqbl\" (UniqueName: \"kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-kube-api-access-zlqbl\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.083717 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.083687    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e250f7a-af92-4a66-9629-b64f6d79326c-istiod-ca-cert\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.083794 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.083722    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-token\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.083794 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.083751    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-certs\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.083874 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.083791    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-credential-socket\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.083912 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.083872    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-podinfo\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.083912 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.083896    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-data\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.083989 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.083976    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-envoy\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.184813 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.184772    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqbl\" (UniqueName: \"kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-kube-api-access-zlqbl\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.185023 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.184845    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e250f7a-af92-4a66-9629-b64f6d79326c-istiod-ca-cert\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.185023 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.184876    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-token\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.185023 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.184904    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-certs\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.185023 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.184952    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-credential-socket\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.185023 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.184983    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-podinfo\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.185023 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.185009    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-data\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.185420 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.185066    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-envoy\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.185486 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.185413    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-socket\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.185680 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.185481    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-credential-socket\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.185680 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.185657    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-certs\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.185896 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.185699    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-socket\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.185896 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.185776    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e250f7a-af92-4a66-9629-b64f6d79326c-istiod-ca-cert\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.186056 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.185907    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-data\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.187692 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.187663    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-envoy\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.187926 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.187906    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-podinfo\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.212801 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.212772    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-token\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.212801 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.212787    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqbl\" (UniqueName: \"kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-kube-api-access-zlqbl\") pod \"e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") " pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.295889 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.295800    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:11.575658 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.575587    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"]
Oct 31 13:10:11.578375 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:10:11.578330    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e250f7a_af92_4a66_9629_b64f6d79326c.slice/crio-43a308798c5f1ca51d421ebc64b2df3a17aef0ce7e4b4fbf0995af003db5b13f WatchSource:0}: Error finding container 43a308798c5f1ca51d421ebc64b2df3a17aef0ce7e4b4fbf0995af003db5b13f: Status 404 returned error can't find the container with id 43a308798c5f1ca51d421ebc64b2df3a17aef0ce7e4b4fbf0995af003db5b13f
Oct 31 13:10:11.580408 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.580374    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:10:11.580475 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.580443    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:10:11.580515 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:11.580475    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:10:12.222669 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:12.222633    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" event={"ID":"6e250f7a-af92-4a66-9629-b64f6d79326c","Type":"ContainerStarted","Data":"e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3"}
Oct 31 13:10:12.223166 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:12.222692    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" event={"ID":"6e250f7a-af92-4a66-9629-b64f6d79326c","Type":"ContainerStarted","Data":"43a308798c5f1ca51d421ebc64b2df3a17aef0ce7e4b4fbf0995af003db5b13f"}
Oct 31 13:10:12.246812 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:12.246764    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" podStartSLOduration=2.24675052 podStartE2EDuration="2.24675052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:12.246207467 +0000 UTC m=+2536.986633684" watchObservedRunningTime="2026-10-31 13:10:12.24675052 +0000 UTC m=+2536.987176736"
Oct 31 13:10:12.296503 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:12.296473    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:12.301497 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:12.301473    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:13.226843 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:13.226813    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:13.227781 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:13.227760    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:15.640727 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.640690    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"]
Oct 31 13:10:15.644124 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.644103    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.656206 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.656176    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"]
Oct 31 13:10:15.695630 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.695598    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-169e4e-serving-cert\""
Oct 31 13:10:15.695630 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.695624    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6jkzx\""
Oct 31 13:10:15.695849 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.695643    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 13:10:15.729817 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.729784    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.729971 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.729821    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqwr\" (UniqueName: \"kubernetes.io/projected/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-kube-api-access-khqwr\") pod \"maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.729971 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.729850    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.730085 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.729977    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.758991 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.758959    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"]
Oct 31 13:10:15.762532 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.762511    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"
Oct 31 13:10:15.775042 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.775010    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"]
Oct 31 13:10:15.775228 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.775197    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-169e4e\""
Oct 31 13:10:15.780077 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.780045    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"]
Oct 31 13:10:15.783528 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.783510    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"
Oct 31 13:10:15.795720 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.795673    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a64d9f3b-dockercfg-rnl8q\""
Oct 31 13:10:15.796918 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.796896    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"]
Oct 31 13:10:15.831268 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.831237    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-169e4e-86d99858559n48\" (UID: \"a892b253-98fa-4b0d-8bb1-4939a3fa4d83\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"
Oct 31 13:10:15.831482 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.831287    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.831482 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.831309    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqwr\" (UniqueName: \"kubernetes.io/projected/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-kube-api-access-khqwr\") pod \"maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.831482 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.831428    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.831689 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.831524    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bebe42-7caa-43fe-a567-df368adf1de1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk\" (UID: \"30bebe42-7caa-43fe-a567-df368adf1de1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"
Oct 31 13:10:15.831689 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.831563    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dlts\" (UniqueName: \"kubernetes.io/projected/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-kube-api-access-7dlts\") pod \"payload-processing-e2e-worker-single-main-169e4e-86d99858559n48\" (UID: \"a892b253-98fa-4b0d-8bb1-4939a3fa4d83\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"
Oct 31 13:10:15.831689 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.831674    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fr64\" (UniqueName: \"kubernetes.io/projected/30bebe42-7caa-43fe-a567-df368adf1de1-kube-api-access-6fr64\") pod \"payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk\" (UID: \"30bebe42-7caa-43fe-a567-df368adf1de1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"
Oct 31 13:10:15.831860 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.831736    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.832483 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.832458    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.833853 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.833833    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.834065 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.834049    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.844101 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.844082    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqwr\" (UniqueName: \"kubernetes.io/projected/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-kube-api-access-khqwr\") pod \"maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:15.933283 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.933245    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fr64\" (UniqueName: \"kubernetes.io/projected/30bebe42-7caa-43fe-a567-df368adf1de1-kube-api-access-6fr64\") pod \"payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk\" (UID: \"30bebe42-7caa-43fe-a567-df368adf1de1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"
Oct 31 13:10:15.933480 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.933336    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-169e4e-86d99858559n48\" (UID: \"a892b253-98fa-4b0d-8bb1-4939a3fa4d83\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"
Oct 31 13:10:15.933480 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.933409    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bebe42-7caa-43fe-a567-df368adf1de1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk\" (UID: \"30bebe42-7caa-43fe-a567-df368adf1de1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"
Oct 31 13:10:15.933480 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.933438    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dlts\" (UniqueName: \"kubernetes.io/projected/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-kube-api-access-7dlts\") pod \"payload-processing-e2e-worker-single-main-169e4e-86d99858559n48\" (UID: \"a892b253-98fa-4b0d-8bb1-4939a3fa4d83\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"
Oct 31 13:10:15.934118 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.934097    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bebe42-7caa-43fe-a567-df368adf1de1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk\" (UID: \"30bebe42-7caa-43fe-a567-df368adf1de1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"
Oct 31 13:10:15.934163 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.934124    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-169e4e-86d99858559n48\" (UID: \"a892b253-98fa-4b0d-8bb1-4939a3fa4d83\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"
Oct 31 13:10:15.945520 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.945492    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dlts\" (UniqueName: \"kubernetes.io/projected/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-kube-api-access-7dlts\") pod \"payload-processing-e2e-worker-single-main-169e4e-86d99858559n48\" (UID: \"a892b253-98fa-4b0d-8bb1-4939a3fa4d83\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"
Oct 31 13:10:15.945655 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.945633    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fr64\" (UniqueName: \"kubernetes.io/projected/30bebe42-7caa-43fe-a567-df368adf1de1-kube-api-access-6fr64\") pod \"payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk\" (UID: \"30bebe42-7caa-43fe-a567-df368adf1de1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"
Oct 31 13:10:15.960320 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:15.960300    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:16.075866 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:16.075835    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"
Oct 31 13:10:16.096265 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:16.096226    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"
Oct 31 13:10:16.237942 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:16.237919    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"]
Oct 31 13:10:16.240045 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:10:16.240019    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9b9091_4a67_4c3a_aa67_d8a90c452ee1.slice/crio-747c5126a337ea63f08e2e3538c3be163547e5e42e75a38ff61be1355c51d05d WatchSource:0}: Error finding container 747c5126a337ea63f08e2e3538c3be163547e5e42e75a38ff61be1355c51d05d: Status 404 returned error can't find the container with id 747c5126a337ea63f08e2e3538c3be163547e5e42e75a38ff61be1355c51d05d
Oct 31 13:10:16.354439 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:16.354414    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"]
Oct 31 13:10:16.356393 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:10:16.356365    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda892b253_98fa_4b0d_8bb1_4939a3fa4d83.slice/crio-ca1afcd4abb1786796febc39b78336a7f839d114277e7947c606ffd843e41cfb WatchSource:0}: Error finding container ca1afcd4abb1786796febc39b78336a7f839d114277e7947c606ffd843e41cfb: Status 404 returned error can't find the container with id ca1afcd4abb1786796febc39b78336a7f839d114277e7947c606ffd843e41cfb
Oct 31 13:10:16.399147 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:16.398901    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"]
Oct 31 13:10:16.404746 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:10:16.404714    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bebe42_7caa_43fe_a567_df368adf1de1.slice/crio-2a6e5a3f9c8bb171d413109a886ff07f88d9e1551fea214074c55531587fc0eb WatchSource:0}: Error finding container 2a6e5a3f9c8bb171d413109a886ff07f88d9e1551fea214074c55531587fc0eb: Status 404 returned error can't find the container with id 2a6e5a3f9c8bb171d413109a886ff07f88d9e1551fea214074c55531587fc0eb
Oct 31 13:10:17.247774 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:17.247693    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48" event={"ID":"a892b253-98fa-4b0d-8bb1-4939a3fa4d83","Type":"ContainerStarted","Data":"dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4"}
Oct 31 13:10:17.247774 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:17.247746    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48" event={"ID":"a892b253-98fa-4b0d-8bb1-4939a3fa4d83","Type":"ContainerStarted","Data":"ca1afcd4abb1786796febc39b78336a7f839d114277e7947c606ffd843e41cfb"}
Oct 31 13:10:17.247774 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:17.247769    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"
Oct 31 13:10:17.249573 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:17.249547    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk" event={"ID":"30bebe42-7caa-43fe-a567-df368adf1de1","Type":"ContainerStarted","Data":"8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5"}
Oct 31 13:10:17.249716 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:17.249591    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk" event={"ID":"30bebe42-7caa-43fe-a567-df368adf1de1","Type":"ContainerStarted","Data":"2a6e5a3f9c8bb171d413109a886ff07f88d9e1551fea214074c55531587fc0eb"}
Oct 31 13:10:17.249716 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:17.249629    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"
Oct 31 13:10:17.251112 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:17.251088    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl" event={"ID":"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1","Type":"ContainerStarted","Data":"17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86"}
Oct 31 13:10:17.251207 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:17.251128    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl" event={"ID":"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1","Type":"ContainerStarted","Data":"747c5126a337ea63f08e2e3538c3be163547e5e42e75a38ff61be1355c51d05d"}
Oct 31 13:10:17.251207 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:17.251150    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:17.269821 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:17.269774    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48" podStartSLOduration=2.269761065 podStartE2EDuration="2.269761065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:17.268314448 +0000 UTC m=+2542.008740664" watchObservedRunningTime="2026-10-31 13:10:17.269761065 +0000 UTC m=+2542.010187324"
Oct 31 13:10:17.294528 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:17.294483    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl" podStartSLOduration=1.64458972 podStartE2EDuration="2.29447137s" totalImagesPullingTime="649.88165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:17.292554148 +0000 UTC m=+2542.032980363" watchObservedRunningTime="2026-10-31 13:10:17.29447137 +0000 UTC m=+2542.034897586"
Oct 31 13:10:22.499823 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:10:22.499771    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:10:23.285463 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:23.285434    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:10:23.306784 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:23.306729    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk" podStartSLOduration=8.306714942 podStartE2EDuration="8.306714942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:17.326033737 +0000 UTC m=+2542.066459954" watchObservedRunningTime="2026-10-31 13:10:23.306714942 +0000 UTC m=+2548.047141184"
Oct 31 13:10:28.227983 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:28.227943    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:10:28.228471 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:28.228027    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 31 13:10:28.262956 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:28.262919    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"
Oct 31 13:10:28.263302 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:28.263280    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"
Oct 31 13:10:28.333537 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:28.333506    5663 generic.go:356] "Generic (PLEG): container finished" podID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerID="e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3" exitCode=0
Oct 31 13:10:28.333710 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:28.333574    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" event={"ID":"6e250f7a-af92-4a66-9629-b64f6d79326c","Type":"ContainerDied","Data":"e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3"}
Oct 31 13:10:28.333938 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:28.333922    5663 scope.go:122] "RemoveContainer" containerID="e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3"
Oct 31 13:10:28.335181 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:28.335144    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:10:28.335520 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:28.335222    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:10:28.335520 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:28.335259    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:10:29.340480 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:29.340451    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" event={"ID":"6e250f7a-af92-4a66-9629-b64f6d79326c","Type":"ContainerStarted","Data":"e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a"}
Oct 31 13:10:30.296011 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:30.295978    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:30.296210 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:30.296190    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:10:30.296263 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:30.296239    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 31 13:10:31.296429 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.296393    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:10:31.296864 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.296449    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 31 13:10:31.486898 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.484154    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"]
Oct 31 13:10:31.488371 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.488328    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.501015 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.500994    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-169e4e\"/\"openshift-service-ca.crt\""
Oct 31 13:10:31.503557 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.503534    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-tmp-dir\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.503659 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.503570    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8583ad2-612a-4879-b2a5-f8bb18751a23-tls-certs\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.503701 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.503685    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-dshm\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.503740 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.503711    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4fl7\" (UniqueName: \"kubernetes.io/projected/f8583ad2-612a-4879-b2a5-f8bb18751a23-kube-api-access-g4fl7\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.503790 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.503766    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-model-cache\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.503882 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.503851    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-home\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.504842 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.504828    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-169e4e\"/\"kube-root-ca.crt\""
Oct 31 13:10:31.507442 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.507421    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-169e4e\"/\"simulator-single-main-169e4e-kserve-self-signed-certs\""
Oct 31 13:10:31.507529 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.507420    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-169e4e\"/\"default-dockercfg-qvkdr\""
Oct 31 13:10:31.513743 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.513657    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"]
Oct 31 13:10:31.604911 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.604876    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-tmp-dir\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.605100 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.604915    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8583ad2-612a-4879-b2a5-f8bb18751a23-tls-certs\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.605100 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.604959    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-dshm\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.605100 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.604978    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4fl7\" (UniqueName: \"kubernetes.io/projected/f8583ad2-612a-4879-b2a5-f8bb18751a23-kube-api-access-g4fl7\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.605100 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.604994    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-model-cache\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.605100 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.605034    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-home\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.605442 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.605410    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-home\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.605734 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.605712    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-model-cache\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.605822 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.605735    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-tmp-dir\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.607738 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.607715    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-dshm\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.608113 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.608090    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8583ad2-612a-4879-b2a5-f8bb18751a23-tls-certs\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.620436 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.620403    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4fl7\" (UniqueName: \"kubernetes.io/projected/f8583ad2-612a-4879-b2a5-f8bb18751a23-kube-api-access-g4fl7\") pod \"simulator-single-main-169e4e-kserve-6787f46b76-kfpxl\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") " pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:31.815695 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:31.815659    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:32.122104 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:32.122074    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"]
Oct 31 13:10:32.124414 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:10:32.124385    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8583ad2_612a_4879_b2a5_f8bb18751a23.slice/crio-4021a58da7ea2c5842baa2acaa9e1a280db3dadd4d38673f6ca3fd59a2622ec1 WatchSource:0}: Error finding container 4021a58da7ea2c5842baa2acaa9e1a280db3dadd4d38673f6ca3fd59a2622ec1: Status 404 returned error can't find the container with id 4021a58da7ea2c5842baa2acaa9e1a280db3dadd4d38673f6ca3fd59a2622ec1
Oct 31 13:10:32.296846 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:32.296814    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:10:32.297255 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:32.296867    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 31 13:10:32.377275 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:32.356593    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl" event={"ID":"f8583ad2-612a-4879-b2a5-f8bb18751a23","Type":"ContainerStarted","Data":"22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7"}
Oct 31 13:10:32.377275 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:32.356641    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl" event={"ID":"f8583ad2-612a-4879-b2a5-f8bb18751a23","Type":"ContainerStarted","Data":"4021a58da7ea2c5842baa2acaa9e1a280db3dadd4d38673f6ca3fd59a2622ec1"}
Oct 31 13:10:32.412337 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:32.412225    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl" podStartSLOduration=1.412211049 podStartE2EDuration="1.412211049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:32.410049311 +0000 UTC m=+2557.150475529" watchObservedRunningTime="2026-10-31 13:10:32.412211049 +0000 UTC m=+2557.152637265"
Oct 31 13:10:32.816576 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:32.816540    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:33.297133 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:33.297051    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 31 13:10:33.297133 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:33.297119    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 31 13:10:34.813661 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:34.813619    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:34.814661 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:34.814614    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:34.815798 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:34.815778    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:10:41.816095 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:41.816053    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:41.828990 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:41.828963    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:42.432317 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:42.432286    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:10:52.476458 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:10:52.476421    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:10:54.023470 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:54.023434    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"]
Oct 31 13:10:54.024766 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:54.024744    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lhvnx"]
Oct 31 13:10:55.438993 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:10:55.438962    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b87f0d5b-4dc2-4f2a-bc0f-428540f395e4" path="/var/lib/kubelet/pods/b87f0d5b-4dc2-4f2a-bc0f-428540f395e4/volumes"
Oct 31 13:11:04.022577 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:04.022540    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"]
Oct 31 13:11:04.025135 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:04.025107    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9g4h2"]
Oct 31 13:11:05.438378 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:05.438331    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36fc0fe4-fd68-4340-b90e-e739b4b3534e" path="/var/lib/kubelet/pods/36fc0fe4-fd68-4340-b90e-e739b4b3534e/volumes"
Oct 31 13:11:17.006489 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:17.006447    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"]
Oct 31 13:11:22.205453 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:22.205409    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"]
Oct 31 13:11:22.227999 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:22.227966    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"]
Oct 31 13:11:22.485397 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:11:22.485280    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:11:23.135161 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:23.135126    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"]
Oct 31 13:11:23.135431 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:23.135379    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl" podUID="ed9b9091-4a67-4c3a-aa67-d8a90c452ee1" containerName="maas-api" containerID="cri-o://17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86" gracePeriod=30
Oct 31 13:11:23.259643 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:23.259609    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl" podUID="ed9b9091-4a67-4c3a-aa67-d8a90c452ee1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.173:8443/health\": dial tcp 10.133.0.173:8443: connect: connection refused"
Oct 31 13:11:24.383362 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.383320    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:11:24.507455 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.507337    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-maas-api-tls\") pod \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") "
Oct 31 13:11:24.507638 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.507474    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-metrics-certs\") pod \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") "
Oct 31 13:11:24.507638 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.507535    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-openshift-service-ca\") pod \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") "
Oct 31 13:11:24.507638 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.507624    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-kube-api-access-khqwr\" (UniqueName: \"kubernetes.io/projected/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-kube-api-access-khqwr\") pod \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\" (UID: \"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1\") "
Oct 31 13:11:24.507900 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.507867    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-openshift-service-ca" pod "ed9b9091-4a67-4c3a-aa67-d8a90c452ee1" (UID: "ed9b9091-4a67-4c3a-aa67-d8a90c452ee1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:11:24.509492 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.509460    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-maas-api-tls" pod "ed9b9091-4a67-4c3a-aa67-d8a90c452ee1" (UID: "ed9b9091-4a67-4c3a-aa67-d8a90c452ee1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:11:24.509595 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.509574    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-metrics-certs" pod "ed9b9091-4a67-4c3a-aa67-d8a90c452ee1" (UID: "ed9b9091-4a67-4c3a-aa67-d8a90c452ee1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:11:24.509656 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.509637    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-kube-api-access-khqwr" pod "ed9b9091-4a67-4c3a-aa67-d8a90c452ee1" (UID: "ed9b9091-4a67-4c3a-aa67-d8a90c452ee1"). InnerVolumeSpecName "kube-api-access-khqwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:11:24.609252 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.609215    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:24.609252 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.609245    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khqwr\" (UniqueName: \"kubernetes.io/projected/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-kube-api-access-khqwr\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:24.609252 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.609257    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:24.609542 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.609266    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:24.620118 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.620089    5663 generic.go:356] "Generic (PLEG): container finished" podID="ed9b9091-4a67-4c3a-aa67-d8a90c452ee1" containerID="17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86" exitCode=0
Oct 31 13:11:24.620254 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.620136    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"
Oct 31 13:11:24.620254 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.620180    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl" event={"ID":"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1","Type":"ContainerDied","Data":"17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86"}
Oct 31 13:11:24.620254 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.620235    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl" event={"ID":"ed9b9091-4a67-4c3a-aa67-d8a90c452ee1","Type":"ContainerDied","Data":"747c5126a337ea63f08e2e3538c3be163547e5e42e75a38ff61be1355c51d05d"}
Oct 31 13:11:24.620379 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.620263    5663 scope.go:122] "RemoveContainer" containerID="17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86"
Oct 31 13:11:24.629813 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.629794    5663 scope.go:122] "RemoveContainer" containerID="17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86"
Oct 31 13:11:24.630108 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:11:24.630089    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86\": container with ID starting with 17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86 not found: ID does not exist" containerID="17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86"
Oct 31 13:11:24.630170 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.630116    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86"} err="failed to get container status \"17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86\": rpc error: code = NotFound desc = could not find container \"17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86\": container with ID starting with 17efe09dabb7809b747d3c65014a024e5a8d19b6cd7f4b7df05ec03d161f2e86 not found: ID does not exist"
Oct 31 13:11:24.641712 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.641692    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"]
Oct 31 13:11:24.645854 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:24.645833    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-169e4e-994d5bd96-8bkcl"]
Oct 31 13:11:25.437832 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:25.437797    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9b9091-4a67-4c3a-aa67-d8a90c452ee1" path="/var/lib/kubelet/pods/ed9b9091-4a67-4c3a-aa67-d8a90c452ee1/volumes"
Oct 31 13:11:27.205812 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:27.205764    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48" podUID="a892b253-98fa-4b0d-8bb1-4939a3fa4d83" containerName="payload-processing" containerID="cri-o://dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4" gracePeriod=25
Oct 31 13:11:27.228448 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:27.228411    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk" podUID="30bebe42-7caa-43fe-a567-df368adf1de1" containerName="payload-pre-processing" containerID="cri-o://8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5" gracePeriod=25
Oct 31 13:11:28.261696 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:28.261650    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48" podUID="a892b253-98fa-4b0d-8bb1-4939a3fa4d83" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:11:28.261696 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:28.261689    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk" podUID="30bebe42-7caa-43fe-a567-df368adf1de1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:11:30.197536 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:30.197484    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"]
Oct 31 13:11:30.198432 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:30.198396    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy" containerID="cri-o://e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a" gracePeriod=30
Oct 31 13:11:32.026009 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.025964    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl" podUID="f8583ad2-612a-4879-b2a5-f8bb18751a23" containerName="main" containerID="cri-o://22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7" gracePeriod=45
Oct 31 13:11:32.268520 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.268496    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:11:32.387732 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.387655    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-model-cache\") pod \"f8583ad2-612a-4879-b2a5-f8bb18751a23\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") "
Oct 31 13:11:32.387732 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.387696    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-dshm\") pod \"f8583ad2-612a-4879-b2a5-f8bb18751a23\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") "
Oct 31 13:11:32.387732 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.387723    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-tmp-dir\") pod \"f8583ad2-612a-4879-b2a5-f8bb18751a23\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") "
Oct 31 13:11:32.387997 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.387782    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-home\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-home\") pod \"f8583ad2-612a-4879-b2a5-f8bb18751a23\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") "
Oct 31 13:11:32.387997 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.387853    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8583ad2-612a-4879-b2a5-f8bb18751a23-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8583ad2-612a-4879-b2a5-f8bb18751a23-tls-certs\") pod \"f8583ad2-612a-4879-b2a5-f8bb18751a23\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") "
Oct 31 13:11:32.387997 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.387891    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8583ad2-612a-4879-b2a5-f8bb18751a23-kube-api-access-g4fl7\" (UniqueName: \"kubernetes.io/projected/f8583ad2-612a-4879-b2a5-f8bb18751a23-kube-api-access-g4fl7\") pod \"f8583ad2-612a-4879-b2a5-f8bb18751a23\" (UID: \"f8583ad2-612a-4879-b2a5-f8bb18751a23\") "
Oct 31 13:11:32.388179 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.387994    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-model-cache" pod "f8583ad2-612a-4879-b2a5-f8bb18751a23" (UID: "f8583ad2-612a-4879-b2a5-f8bb18751a23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:11:32.388179 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.388019    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-tmp-dir" pod "f8583ad2-612a-4879-b2a5-f8bb18751a23" (UID: "f8583ad2-612a-4879-b2a5-f8bb18751a23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:11:32.388179 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.388111    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-home" pod "f8583ad2-612a-4879-b2a5-f8bb18751a23" (UID: "f8583ad2-612a-4879-b2a5-f8bb18751a23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:11:32.388367 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.388266    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:32.388367 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.388311    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:32.388367 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.388328    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:32.389934 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.389906    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-dshm" pod "f8583ad2-612a-4879-b2a5-f8bb18751a23" (UID: "f8583ad2-612a-4879-b2a5-f8bb18751a23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:11:32.390013 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.389962    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8583ad2-612a-4879-b2a5-f8bb18751a23-kube-api-access-g4fl7" pod "f8583ad2-612a-4879-b2a5-f8bb18751a23" (UID: "f8583ad2-612a-4879-b2a5-f8bb18751a23"). InnerVolumeSpecName "kube-api-access-g4fl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:11:32.390065 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.390010    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8583ad2-612a-4879-b2a5-f8bb18751a23-tls-certs" pod "f8583ad2-612a-4879-b2a5-f8bb18751a23" (UID: "f8583ad2-612a-4879-b2a5-f8bb18751a23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:11:32.489214 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.489174    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8583ad2-612a-4879-b2a5-f8bb18751a23-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:32.489214 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.489206    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8583ad2-612a-4879-b2a5-f8bb18751a23-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:32.489214 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.489215    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4fl7\" (UniqueName: \"kubernetes.io/projected/f8583ad2-612a-4879-b2a5-f8bb18751a23-kube-api-access-g4fl7\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:32.659925 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.659847    5663 generic.go:356] "Generic (PLEG): container finished" podID="f8583ad2-612a-4879-b2a5-f8bb18751a23" containerID="22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7" exitCode=0
Oct 31 13:11:32.659925 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.659899    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"
Oct 31 13:11:32.660115 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.659930    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl" event={"ID":"f8583ad2-612a-4879-b2a5-f8bb18751a23","Type":"ContainerDied","Data":"22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7"}
Oct 31 13:11:32.660115 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.659984    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl" event={"ID":"f8583ad2-612a-4879-b2a5-f8bb18751a23","Type":"ContainerDied","Data":"4021a58da7ea2c5842baa2acaa9e1a280db3dadd4d38673f6ca3fd59a2622ec1"}
Oct 31 13:11:32.660115 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.660007    5663 scope.go:122] "RemoveContainer" containerID="22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7"
Oct 31 13:11:32.669663 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.669640    5663 scope.go:122] "RemoveContainer" containerID="22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7"
Oct 31 13:11:32.669920 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:11:32.669898    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7\": container with ID starting with 22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7 not found: ID does not exist" containerID="22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7"
Oct 31 13:11:32.669974 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.669929    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7"} err="failed to get container status \"22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7\": rpc error: code = NotFound desc = could not find container \"22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7\": container with ID starting with 22a2e6c8eed5a5ad933372bfe3cd9ff65ef969ac6beb50b2c099fcbfb4dd03a7 not found: ID does not exist"
Oct 31 13:11:32.682165 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.682141    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"]
Oct 31 13:11:32.686070 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:32.686042    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-169e4e/simulator-single-main-169e4e-kserve-6787f46b76-kfpxl"]
Oct 31 13:11:33.438185 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:33.438145    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8583ad2-612a-4879-b2a5-f8bb18751a23" path="/var/lib/kubelet/pods/f8583ad2-612a-4879-b2a5-f8bb18751a23/volumes"
Oct 31 13:11:34.710673 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.710638    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"]
Oct 31 13:11:34.711106 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.710998    5663 cpu_manager.go:414] "containerMap: removing container" podUID="f8583ad2-612a-4879-b2a5-f8bb18751a23" containerName="main"
Oct 31 13:11:34.711106 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.711010    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8583ad2-612a-4879-b2a5-f8bb18751a23" containerName="main"
Oct 31 13:11:34.711106 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.711031    5663 cpu_manager.go:414] "containerMap: removing container" podUID="ed9b9091-4a67-4c3a-aa67-d8a90c452ee1" containerName="maas-api"
Oct 31 13:11:34.711106 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.711046    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9b9091-4a67-4c3a-aa67-d8a90c452ee1" containerName="maas-api"
Oct 31 13:11:34.711262 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.711122    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9b9091-4a67-4c3a-aa67-d8a90c452ee1" containerName="maas-api"
Oct 31 13:11:34.711262 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.711137    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8583ad2-612a-4879-b2a5-f8bb18751a23" containerName="main"
Oct 31 13:11:34.716400 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.716348    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.724252 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.724226    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"]
Oct 31 13:11:34.728896 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.728865    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2xp25\""
Oct 31 13:11:34.729519 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.729486    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-26740100-kserve-self-signed-certs\""
Oct 31 13:11:34.729675 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.728870    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 13:11:34.729675 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.729770    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 13:11:34.811623 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.811592    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-dshm\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.811906 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.811646    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-home\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.811906 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.811726    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-tmp-dir\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.811906 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.811776    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-model-cache\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.812089 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.811914    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd629212-3415-46ab-a6c9-ca7518a1925e-tls-certs\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.812089 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.811949    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x4z2\" (UniqueName: \"kubernetes.io/projected/dd629212-3415-46ab-a6c9-ca7518a1925e-kube-api-access-5x4z2\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.816963 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.816933    5663 patch_prober.go:29] interesting pod/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:11:34.817114 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.816991    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:11:34.912673 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.912639    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd629212-3415-46ab-a6c9-ca7518a1925e-tls-certs\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.912673 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.912675    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x4z2\" (UniqueName: \"kubernetes.io/projected/dd629212-3415-46ab-a6c9-ca7518a1925e-kube-api-access-5x4z2\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.912906 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.912800    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-dshm\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.912906 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.912885    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-home\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.913059 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.912923    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-tmp-dir\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.913059 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.912972    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-model-cache\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.913437 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.913410    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-home\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.913437 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.913430    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-model-cache\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.913624 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.913499    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-tmp-dir\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.915174 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.915151    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-dshm\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.915456 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.915439    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd629212-3415-46ab-a6c9-ca7518a1925e-tls-certs\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:34.924880 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:34.924860    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x4z2\" (UniqueName: \"kubernetes.io/projected/dd629212-3415-46ab-a6c9-ca7518a1925e-kube-api-access-5x4z2\") pod \"e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") " pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:35.040135 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.040053    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:35.317348 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.317316    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"]
Oct 31 13:11:35.320117 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:11:35.320091    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd629212_3415_46ab_a6c9_ca7518a1925e.slice/crio-9e606c5f5069a141cfac044b3dd6aaa54367732624d7dd13a8134b8d9c778cd0 WatchSource:0}: Error finding container 9e606c5f5069a141cfac044b3dd6aaa54367732624d7dd13a8134b8d9c778cd0: Status 404 returned error can't find the container with id 9e606c5f5069a141cfac044b3dd6aaa54367732624d7dd13a8134b8d9c778cd0
Oct 31 13:11:35.446526 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.446500    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:11:35.619038 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.618944    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-podinfo\") pod \"6e250f7a-af92-4a66-9629-b64f6d79326c\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") "
Oct 31 13:11:35.619038 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.618994    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-socket\") pod \"6e250f7a-af92-4a66-9629-b64f6d79326c\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") "
Oct 31 13:11:35.619038 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619036    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e250f7a-af92-4a66-9629-b64f6d79326c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e250f7a-af92-4a66-9629-b64f6d79326c-istiod-ca-cert\") pod \"6e250f7a-af92-4a66-9629-b64f6d79326c\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") "
Oct 31 13:11:35.619335 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619081    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-data\") pod \"6e250f7a-af92-4a66-9629-b64f6d79326c\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") "
Oct 31 13:11:35.619335 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619111    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-credential-socket\") pod \"6e250f7a-af92-4a66-9629-b64f6d79326c\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") "
Oct 31 13:11:35.619335 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619153    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-certs\") pod \"6e250f7a-af92-4a66-9629-b64f6d79326c\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") "
Oct 31 13:11:35.619335 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619201    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-kube-api-access-zlqbl\" (UniqueName: \"kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-kube-api-access-zlqbl\") pod \"6e250f7a-af92-4a66-9629-b64f6d79326c\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") "
Oct 31 13:11:35.619335 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619235    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-envoy\") pod \"6e250f7a-af92-4a66-9629-b64f6d79326c\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") "
Oct 31 13:11:35.619335 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619268    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-token\" (UniqueName: \"kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-token\") pod \"6e250f7a-af92-4a66-9629-b64f6d79326c\" (UID: \"6e250f7a-af92-4a66-9629-b64f6d79326c\") "
Oct 31 13:11:35.619671 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619349    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-socket" pod "6e250f7a-af92-4a66-9629-b64f6d79326c" (UID: "6e250f7a-af92-4a66-9629-b64f6d79326c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:11:35.619671 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619567    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-certs" pod "6e250f7a-af92-4a66-9629-b64f6d79326c" (UID: "6e250f7a-af92-4a66-9629-b64f6d79326c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:11:35.619799 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619683    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e250f7a-af92-4a66-9629-b64f6d79326c-istiod-ca-cert" pod "6e250f7a-af92-4a66-9629-b64f6d79326c" (UID: "6e250f7a-af92-4a66-9629-b64f6d79326c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:11:35.619799 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619692    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:35.619799 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619728    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:35.620023 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619922    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-credential-socket" pod "6e250f7a-af92-4a66-9629-b64f6d79326c" (UID: "6e250f7a-af92-4a66-9629-b64f6d79326c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:11:35.620023 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.619988    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-data" pod "6e250f7a-af92-4a66-9629-b64f6d79326c" (UID: "6e250f7a-af92-4a66-9629-b64f6d79326c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:11:35.621909 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.621884    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-token" pod "6e250f7a-af92-4a66-9629-b64f6d79326c" (UID: "6e250f7a-af92-4a66-9629-b64f6d79326c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:11:35.622046 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.621884    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-podinfo" pod "6e250f7a-af92-4a66-9629-b64f6d79326c" (UID: "6e250f7a-af92-4a66-9629-b64f6d79326c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:11:35.622046 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.621990    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-kube-api-access-zlqbl" pod "6e250f7a-af92-4a66-9629-b64f6d79326c" (UID: "6e250f7a-af92-4a66-9629-b64f6d79326c"). InnerVolumeSpecName "kube-api-access-zlqbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:11:35.622046 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.621987    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-envoy" pod "6e250f7a-af92-4a66-9629-b64f6d79326c" (UID: "6e250f7a-af92-4a66-9629-b64f6d79326c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:11:35.675958 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.675923    5663 generic.go:356] "Generic (PLEG): container finished" podID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerID="e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a" exitCode=0
Oct 31 13:11:35.676126 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.675971    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"
Oct 31 13:11:35.676126 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.676000    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" event={"ID":"6e250f7a-af92-4a66-9629-b64f6d79326c","Type":"ContainerDied","Data":"e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a"}
Oct 31 13:11:35.676126 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.676063    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt" event={"ID":"6e250f7a-af92-4a66-9629-b64f6d79326c","Type":"ContainerDied","Data":"43a308798c5f1ca51d421ebc64b2df3a17aef0ce7e4b4fbf0995af003db5b13f"}
Oct 31 13:11:35.676126 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.676095    5663 scope.go:122] "RemoveContainer" containerID="e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a"
Oct 31 13:11:35.677871 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.677843    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg" event={"ID":"dd629212-3415-46ab-a6c9-ca7518a1925e","Type":"ContainerStarted","Data":"7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460"}
Oct 31 13:11:35.678002 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.677883    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg" event={"ID":"dd629212-3415-46ab-a6c9-ca7518a1925e","Type":"ContainerStarted","Data":"9e606c5f5069a141cfac044b3dd6aaa54367732624d7dd13a8134b8d9c778cd0"}
Oct 31 13:11:35.685909 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.685891    5663 scope.go:122] "RemoveContainer" containerID="e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3"
Oct 31 13:11:35.695338 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.695321    5663 scope.go:122] "RemoveContainer" containerID="e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a"
Oct 31 13:11:35.695636 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:11:35.695613    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a\": container with ID starting with e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a not found: ID does not exist" containerID="e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a"
Oct 31 13:11:35.695689 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.695650    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a"} err="failed to get container status \"e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a\": rpc error: code = NotFound desc = could not find container \"e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a\": container with ID starting with e62f4af6a471f570e8c24bf51e2f3ed16bcba0791adfe38b8f0dd965bcde996a not found: ID does not exist"
Oct 31 13:11:35.695689 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.695673    5663 scope.go:122] "RemoveContainer" containerID="e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3"
Oct 31 13:11:35.695924 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:11:35.695905    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3\": container with ID starting with e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3 not found: ID does not exist" containerID="e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3"
Oct 31 13:11:35.695969 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.695935    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3"} err="failed to get container status \"e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3\": rpc error: code = NotFound desc = could not find container \"e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3\": container with ID starting with e9ce1cd97bebe6adcd4b1b35b8db701dd3e4d9ca55c6674289e3109f1a4381c3 not found: ID does not exist"
Oct 31 13:11:35.701894 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.701757    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg" podStartSLOduration=1.7017448769999999 podStartE2EDuration="1.701744877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:35.700024451 +0000 UTC m=+2620.440450668" watchObservedRunningTime="2026-10-31 13:11:35.701744877 +0000 UTC m=+2620.442171107"
Oct 31 13:11:35.715493 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.715469    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"]
Oct 31 13:11:35.719531 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.719510    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-169e4e-openshift-default-d9d58bd85-xqwlt"]
Oct 31 13:11:35.720607 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.720589    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:35.720667 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.720609    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e250f7a-af92-4a66-9629-b64f6d79326c-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:35.720667 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.720620    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:35.720667 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.720629    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:35.720667 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.720638    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlqbl\" (UniqueName: \"kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-kube-api-access-zlqbl\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:35.720667 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.720647    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:35.720667 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:35.720655    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e250f7a-af92-4a66-9629-b64f6d79326c-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:36.041191 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:36.041156    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:37.440611 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:37.440562    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" path="/var/lib/kubelet/pods/6e250f7a-af92-4a66-9629-b64f6d79326c/volumes"
Oct 31 13:11:38.276378 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:38.275444    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48" podUID="a892b253-98fa-4b0d-8bb1-4939a3fa4d83" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:11:38.276605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:38.276436    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk" podUID="30bebe42-7caa-43fe-a567-df368adf1de1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:11:45.040807 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:45.040765    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:45.058704 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:45.058673    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:45.747184 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:45.747155    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:11:47.468740 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.468688    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"
Oct 31 13:11:47.501592 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.501571    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"
Oct 31 13:11:47.555474 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.555447    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-plugins-config-volume\") pod \"a892b253-98fa-4b0d-8bb1-4939a3fa4d83\" (UID: \"a892b253-98fa-4b0d-8bb1-4939a3fa4d83\") "
Oct 31 13:11:47.555664 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.555507    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-kube-api-access-7dlts\" (UniqueName: \"kubernetes.io/projected/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-kube-api-access-7dlts\") pod \"a892b253-98fa-4b0d-8bb1-4939a3fa4d83\" (UID: \"a892b253-98fa-4b0d-8bb1-4939a3fa4d83\") "
Oct 31 13:11:47.555888 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.555858    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-plugins-config-volume" pod "a892b253-98fa-4b0d-8bb1-4939a3fa4d83" (UID: "a892b253-98fa-4b0d-8bb1-4939a3fa4d83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:11:47.557694 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.557672    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-kube-api-access-7dlts" pod "a892b253-98fa-4b0d-8bb1-4939a3fa4d83" (UID: "a892b253-98fa-4b0d-8bb1-4939a3fa4d83"). InnerVolumeSpecName "kube-api-access-7dlts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:11:47.656219 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.656187    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30bebe42-7caa-43fe-a567-df368adf1de1-kube-api-access-6fr64\" (UniqueName: \"kubernetes.io/projected/30bebe42-7caa-43fe-a567-df368adf1de1-kube-api-access-6fr64\") pod \"30bebe42-7caa-43fe-a567-df368adf1de1\" (UID: \"30bebe42-7caa-43fe-a567-df368adf1de1\") "
Oct 31 13:11:47.656392 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.656285    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30bebe42-7caa-43fe-a567-df368adf1de1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bebe42-7caa-43fe-a567-df368adf1de1-plugins-config-volume\") pod \"30bebe42-7caa-43fe-a567-df368adf1de1\" (UID: \"30bebe42-7caa-43fe-a567-df368adf1de1\") "
Oct 31 13:11:47.656462 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.656449    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:47.656501 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.656465    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dlts\" (UniqueName: \"kubernetes.io/projected/a892b253-98fa-4b0d-8bb1-4939a3fa4d83-kube-api-access-7dlts\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:47.656655 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.656625    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bebe42-7caa-43fe-a567-df368adf1de1-plugins-config-volume" pod "30bebe42-7caa-43fe-a567-df368adf1de1" (UID: "30bebe42-7caa-43fe-a567-df368adf1de1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:11:47.658240 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.658222    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bebe42-7caa-43fe-a567-df368adf1de1-kube-api-access-6fr64" pod "30bebe42-7caa-43fe-a567-df368adf1de1" (UID: "30bebe42-7caa-43fe-a567-df368adf1de1"). InnerVolumeSpecName "kube-api-access-6fr64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:11:47.744989 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.744902    5663 generic.go:356] "Generic (PLEG): container finished" podID="a892b253-98fa-4b0d-8bb1-4939a3fa4d83" containerID="dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4" exitCode=0
Oct 31 13:11:47.744989 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.744949    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"
Oct 31 13:11:47.745229 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.744982    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48" event={"ID":"a892b253-98fa-4b0d-8bb1-4939a3fa4d83","Type":"ContainerDied","Data":"dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4"}
Oct 31 13:11:47.745229 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.745032    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48" event={"ID":"a892b253-98fa-4b0d-8bb1-4939a3fa4d83","Type":"ContainerDied","Data":"ca1afcd4abb1786796febc39b78336a7f839d114277e7947c606ffd843e41cfb"}
Oct 31 13:11:47.745229 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.745055    5663 scope.go:122] "RemoveContainer" containerID="dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4"
Oct 31 13:11:47.746716 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.746688    5663 generic.go:356] "Generic (PLEG): container finished" podID="30bebe42-7caa-43fe-a567-df368adf1de1" containerID="8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5" exitCode=0
Oct 31 13:11:47.746834 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.746721    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk" event={"ID":"30bebe42-7caa-43fe-a567-df368adf1de1","Type":"ContainerDied","Data":"8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5"}
Oct 31 13:11:47.746834 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.746755    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"
Oct 31 13:11:47.746834 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.746776    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk" event={"ID":"30bebe42-7caa-43fe-a567-df368adf1de1","Type":"ContainerDied","Data":"2a6e5a3f9c8bb171d413109a886ff07f88d9e1551fea214074c55531587fc0eb"}
Oct 31 13:11:47.755062 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.755038    5663 scope.go:122] "RemoveContainer" containerID="dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4"
Oct 31 13:11:47.755326 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:11:47.755306    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4\": container with ID starting with dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4 not found: ID does not exist" containerID="dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4"
Oct 31 13:11:47.755443 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.755333    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4"} err="failed to get container status \"dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4\": rpc error: code = NotFound desc = could not find container \"dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4\": container with ID starting with dce9218da283b7582276db530009fa717b7128597b83999d93c099fc62e74fe4 not found: ID does not exist"
Oct 31 13:11:47.755443 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.755366    5663 scope.go:122] "RemoveContainer" containerID="8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5"
Oct 31 13:11:47.756929 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.756872    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bebe42-7caa-43fe-a567-df368adf1de1-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:47.756929 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.756900    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fr64\" (UniqueName: \"kubernetes.io/projected/30bebe42-7caa-43fe-a567-df368adf1de1-kube-api-access-6fr64\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:11:47.762980 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.762964    5663 scope.go:122] "RemoveContainer" containerID="8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5"
Oct 31 13:11:47.763208 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:11:47.763192    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5\": container with ID starting with 8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5 not found: ID does not exist" containerID="8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5"
Oct 31 13:11:47.763258 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.763213    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5"} err="failed to get container status \"8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5\": rpc error: code = NotFound desc = could not find container \"8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5\": container with ID starting with 8e6af1fba7e7770ec6898aa9c549c0b23607b86ff7707b19e7f0d3991b6a63c5 not found: ID does not exist"
Oct 31 13:11:47.770237 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.770212    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"]
Oct 31 13:11:47.774455 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.774436    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-169e4e-86d99858559n48"]
Oct 31 13:11:47.785469 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.785450    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"]
Oct 31 13:11:47.789643 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:47.789623    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-169e4e-5f869nbtjk"]
Oct 31 13:11:49.437521 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:49.437488    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30bebe42-7caa-43fe-a567-df368adf1de1" path="/var/lib/kubelet/pods/30bebe42-7caa-43fe-a567-df368adf1de1/volumes"
Oct 31 13:11:49.437917 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:49.437855    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a892b253-98fa-4b0d-8bb1-4939a3fa4d83" path="/var/lib/kubelet/pods/a892b253-98fa-4b0d-8bb1-4939a3fa4d83/volumes"
Oct 31 13:11:50.047028 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.046993    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"]
Oct 31 13:11:50.047361 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.047337    5663 cpu_manager.go:414] "containerMap: removing container" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy"
Oct 31 13:11:50.047415 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.047365    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy"
Oct 31 13:11:50.047415 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.047380    5663 cpu_manager.go:414] "containerMap: removing container" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy"
Oct 31 13:11:50.047415 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.047385    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy"
Oct 31 13:11:50.047575 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.047442    5663 cpu_manager.go:414] "containerMap: removing container" podUID="30bebe42-7caa-43fe-a567-df368adf1de1" containerName="payload-pre-processing"
Oct 31 13:11:50.047575 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.047451    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30bebe42-7caa-43fe-a567-df368adf1de1" containerName="payload-pre-processing"
Oct 31 13:11:50.047575 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.047469    5663 cpu_manager.go:414] "containerMap: removing container" podUID="a892b253-98fa-4b0d-8bb1-4939a3fa4d83" containerName="payload-processing"
Oct 31 13:11:50.047575 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.047476    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a892b253-98fa-4b0d-8bb1-4939a3fa4d83" containerName="payload-processing"
Oct 31 13:11:50.047744 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.047590    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="30bebe42-7caa-43fe-a567-df368adf1de1" containerName="payload-pre-processing"
Oct 31 13:11:50.047744 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.047606    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="a892b253-98fa-4b0d-8bb1-4939a3fa4d83" containerName="payload-processing"
Oct 31 13:11:50.047744 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.047617    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy"
Oct 31 13:11:50.047744 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.047626    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e250f7a-af92-4a66-9629-b64f6d79326c" containerName="istio-proxy"
Oct 31 13:11:50.051051 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.051033    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.058514 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.058494    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-26740100-kserve-self-signed-certs\""
Oct 31 13:11:50.068226 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.068196    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"]
Oct 31 13:11:50.180207 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.180169    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-model-cache\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.180444 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.180230    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-home\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.180444 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.180411    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e00fbee-619e-43c1-ab75-b468f1bae823-tls-certs\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.180570 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.180460    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw8n2\" (UniqueName: \"kubernetes.io/projected/8e00fbee-619e-43c1-ab75-b468f1bae823-kube-api-access-qw8n2\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.180570 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.180556    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-tmp-dir\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.180690 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.180616    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-dshm\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.281566 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.281525    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-dshm\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.281769 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.281591    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-model-cache\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.281769 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.281624    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-home\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.281769 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.281676    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e00fbee-619e-43c1-ab75-b468f1bae823-tls-certs\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.281769 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.281709    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8n2\" (UniqueName: \"kubernetes.io/projected/8e00fbee-619e-43c1-ab75-b468f1bae823-kube-api-access-qw8n2\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.281769 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.281748    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-tmp-dir\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.282128 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.282102    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-model-cache\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.282218 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.282130    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-home\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.282218 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.282204    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-tmp-dir\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.283898 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.283876    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-dshm\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.284197 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.284179    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e00fbee-619e-43c1-ab75-b468f1bae823-tls-certs\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.294458 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.294430    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw8n2\" (UniqueName: \"kubernetes.io/projected/8e00fbee-619e-43c1-ab75-b468f1bae823-kube-api-access-qw8n2\") pod \"e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") " pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.362676 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.362639    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:50.638177 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.637983    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"]
Oct 31 13:11:50.640834 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:11:50.640806    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e00fbee_619e_43c1_ab75_b468f1bae823.slice/crio-fab79e726b3f1472c9b2e678c79c925caedc700f18da51df2a996d7051e19975 WatchSource:0}: Error finding container fab79e726b3f1472c9b2e678c79c925caedc700f18da51df2a996d7051e19975: Status 404 returned error can't find the container with id fab79e726b3f1472c9b2e678c79c925caedc700f18da51df2a996d7051e19975
Oct 31 13:11:50.763978 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.763941    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc" event={"ID":"8e00fbee-619e-43c1-ab75-b468f1bae823","Type":"ContainerStarted","Data":"1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174"}
Oct 31 13:11:50.764195 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.763986    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc" event={"ID":"8e00fbee-619e-43c1-ab75-b468f1bae823","Type":"ContainerStarted","Data":"fab79e726b3f1472c9b2e678c79c925caedc700f18da51df2a996d7051e19975"}
Oct 31 13:11:50.786267 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:50.786221    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc" podStartSLOduration=0.786208139 podStartE2EDuration="786.208139ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:50.785720299 +0000 UTC m=+2635.526146516" watchObservedRunningTime="2026-10-31 13:11:50.786208139 +0000 UTC m=+2635.526634358"
Oct 31 13:11:51.362920 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:51.362887    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:11:52.477070 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:11:52.477026    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:11:59.753135 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:11:59.753099    5663 scope.go:122] "RemoveContainer" containerID="efa0bc577f846034757a3a5b9e0fa09d58845b6c78238a5435cd4456dd33789c"
Oct 31 13:12:00.363155 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:00.363119    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:12:00.375652 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:00.375619    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:12:00.843784 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:00.843758    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:12:07.472507 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.472466    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"]
Oct 31 13:12:07.477635 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.477612    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:07.485064 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.485039    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:12:07.485064 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.485048    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:12:07.487051 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.487023    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"]
Oct 31 13:12:07.566071 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.566028    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-25pvx\" (UID: \"c786f506-d163-4051-8d83-27f982c1c9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:07.566298 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.566085    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6pj\" (UniqueName: \"kubernetes.io/projected/c786f506-d163-4051-8d83-27f982c1c9ea-kube-api-access-gn6pj\") pod \"maas-api-revoke-sub-452b6d7b-25pvx\" (UID: \"c786f506-d163-4051-8d83-27f982c1c9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:07.566298 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.566235    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-25pvx\" (UID: \"c786f506-d163-4051-8d83-27f982c1c9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:07.667500 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.667448    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-25pvx\" (UID: \"c786f506-d163-4051-8d83-27f982c1c9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:07.667707 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.667517    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6pj\" (UniqueName: \"kubernetes.io/projected/c786f506-d163-4051-8d83-27f982c1c9ea-kube-api-access-gn6pj\") pod \"maas-api-revoke-sub-452b6d7b-25pvx\" (UID: \"c786f506-d163-4051-8d83-27f982c1c9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:07.667707 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.667578    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-25pvx\" (UID: \"c786f506-d163-4051-8d83-27f982c1c9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:07.668295 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.668264    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-25pvx\" (UID: \"c786f506-d163-4051-8d83-27f982c1c9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:07.670191 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.670153    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-25pvx\" (UID: \"c786f506-d163-4051-8d83-27f982c1c9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:07.680876 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.680845    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6pj\" (UniqueName: \"kubernetes.io/projected/c786f506-d163-4051-8d83-27f982c1c9ea-kube-api-access-gn6pj\") pod \"maas-api-revoke-sub-452b6d7b-25pvx\" (UID: \"c786f506-d163-4051-8d83-27f982c1c9ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:07.791389 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:07.791275    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:08.064380 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:08.064324    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"]
Oct 31 13:12:08.067382 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:12:08.067320    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc786f506_d163_4051_8d83_27f982c1c9ea.slice/crio-ad4ae2d78fd854d5fa90641275a37433c370a417c9b537709ae4784ae2a17071 WatchSource:0}: Error finding container ad4ae2d78fd854d5fa90641275a37433c370a417c9b537709ae4784ae2a17071: Status 404 returned error can't find the container with id ad4ae2d78fd854d5fa90641275a37433c370a417c9b537709ae4784ae2a17071
Oct 31 13:12:08.871145 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:08.871102    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx" event={"ID":"c786f506-d163-4051-8d83-27f982c1c9ea","Type":"ContainerStarted","Data":"11e87ed4d20b662370df5ed038b76e4a4ea04d249ee4882218f1064e178b5b35"}
Oct 31 13:12:08.871145 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:08.871167    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx" event={"ID":"c786f506-d163-4051-8d83-27f982c1c9ea","Type":"ContainerStarted","Data":"ad4ae2d78fd854d5fa90641275a37433c370a417c9b537709ae4784ae2a17071"}
Oct 31 13:12:08.891958 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:08.891905    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx" podStartSLOduration=1.891891185 podStartE2EDuration="1.891891185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:08.889020876 +0000 UTC m=+2653.629447091" watchObservedRunningTime="2026-10-31 13:12:08.891891185 +0000 UTC m=+2653.632317402"
Oct 31 13:12:09.929061 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:09.929016    5663 generic.go:356] "Generic (PLEG): container finished" podID="c786f506-d163-4051-8d83-27f982c1c9ea" containerID="11e87ed4d20b662370df5ed038b76e4a4ea04d249ee4882218f1064e178b5b35" exitCode=0
Oct 31 13:12:09.929594 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:09.929085    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx" event={"ID":"c786f506-d163-4051-8d83-27f982c1c9ea","Type":"ContainerDied","Data":"11e87ed4d20b662370df5ed038b76e4a4ea04d249ee4882218f1064e178b5b35"}
Oct 31 13:12:10.856532 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:10.856492    5663 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-dswld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:12:10.856828 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:10.856798    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-dswld" podUID="57ad591e-3b45-4312-8612-cf848dcf55e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:12:11.288379 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.287579    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:11.345091 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.345053    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c786f506-d163-4051-8d83-27f982c1c9ea-kube-api-access-gn6pj\" (UniqueName: \"kubernetes.io/projected/c786f506-d163-4051-8d83-27f982c1c9ea-kube-api-access-gn6pj\") pod \"c786f506-d163-4051-8d83-27f982c1c9ea\" (UID: \"c786f506-d163-4051-8d83-27f982c1c9ea\") "
Oct 31 13:12:11.345265 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.345104    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-openshift-ca\") pod \"c786f506-d163-4051-8d83-27f982c1c9ea\" (UID: \"c786f506-d163-4051-8d83-27f982c1c9ea\") "
Oct 31 13:12:11.345265 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.345193    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-xks-ca\") pod \"c786f506-d163-4051-8d83-27f982c1c9ea\" (UID: \"c786f506-d163-4051-8d83-27f982c1c9ea\") "
Oct 31 13:12:11.345815 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.345782    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-openshift-ca" pod "c786f506-d163-4051-8d83-27f982c1c9ea" (UID: "c786f506-d163-4051-8d83-27f982c1c9ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:11.347273 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.347251    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-xks-ca" pod "c786f506-d163-4051-8d83-27f982c1c9ea" (UID: "c786f506-d163-4051-8d83-27f982c1c9ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:11.347401 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.347315    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c786f506-d163-4051-8d83-27f982c1c9ea-kube-api-access-gn6pj" pod "c786f506-d163-4051-8d83-27f982c1c9ea" (UID: "c786f506-d163-4051-8d83-27f982c1c9ea"). InnerVolumeSpecName "kube-api-access-gn6pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:11.446213 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.446066    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn6pj\" (UniqueName: \"kubernetes.io/projected/c786f506-d163-4051-8d83-27f982c1c9ea-kube-api-access-gn6pj\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:11.446213 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.446094    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:11.446213 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.446109    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786f506-d163-4051-8d83-27f982c1c9ea-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:11.942405 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.942372    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"
Oct 31 13:12:11.942405 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.942394    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx" event={"ID":"c786f506-d163-4051-8d83-27f982c1c9ea","Type":"ContainerDied","Data":"ad4ae2d78fd854d5fa90641275a37433c370a417c9b537709ae4784ae2a17071"}
Oct 31 13:12:11.942696 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:11.942443    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad4ae2d78fd854d5fa90641275a37433c370a417c9b537709ae4784ae2a17071"
Oct 31 13:12:18.280670 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:18.280632    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"]
Oct 31 13:12:20.531944 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:20.531889    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"]
Oct 31 13:12:22.552391 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:12:22.552317    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:12:24.071128 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.071085    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"]
Oct 31 13:12:24.071647 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.071607    5663 cpu_manager.go:414] "containerMap: removing container" podUID="c786f506-d163-4051-8d83-27f982c1c9ea" containerName="revoke-keys"
Oct 31 13:12:24.071647 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.071626    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c786f506-d163-4051-8d83-27f982c1c9ea" containerName="revoke-keys"
Oct 31 13:12:24.071766 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.071725    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="c786f506-d163-4051-8d83-27f982c1c9ea" containerName="revoke-keys"
Oct 31 13:12:24.074964 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.074940    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:24.085285 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.085265    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:12:24.089308 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.089288    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:12:24.096054 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.096028    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"]
Oct 31 13:12:24.177365 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.177326    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8khgk\" (UniqueName: \"kubernetes.io/projected/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-kube-api-access-8khgk\") pod \"maas-api-revoke-sub-3e31a506-8zvrv\" (UID: \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:24.177534 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.177446    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8zvrv\" (UID: \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:24.177534 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.177489    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8zvrv\" (UID: \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:24.278672 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.278638    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8zvrv\" (UID: \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:24.278888 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.278739    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8khgk\" (UniqueName: \"kubernetes.io/projected/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-kube-api-access-8khgk\") pod \"maas-api-revoke-sub-3e31a506-8zvrv\" (UID: \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:24.278888 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.278805    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8zvrv\" (UID: \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:24.279546 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.279446    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8zvrv\" (UID: \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:24.281185 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.281160    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8zvrv\" (UID: \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:24.292121 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.292079    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8khgk\" (UniqueName: \"kubernetes.io/projected/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-kube-api-access-8khgk\") pod \"maas-api-revoke-sub-3e31a506-8zvrv\" (UID: \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:24.388020 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.387926    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:24.674655 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:24.674621    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"]
Oct 31 13:12:24.678122 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:12:24.678088    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a257c14_3432_4628_b2e2_e2f3cf9f18c1.slice/crio-a6c89eac32c250fb0717551c5fdc0142ba230f157fad1f9c114da3abae5df0da WatchSource:0}: Error finding container a6c89eac32c250fb0717551c5fdc0142ba230f157fad1f9c114da3abae5df0da: Status 404 returned error can't find the container with id a6c89eac32c250fb0717551c5fdc0142ba230f157fad1f9c114da3abae5df0da
Oct 31 13:12:25.043515 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:25.043407    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv" event={"ID":"8a257c14-3432-4628-b2e2-e2f3cf9f18c1","Type":"ContainerStarted","Data":"2c50f408782ac1c3099f345103f6f646ee8548c44c56006f1387bfad0052464d"}
Oct 31 13:12:25.043515 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:25.043472    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv" event={"ID":"8a257c14-3432-4628-b2e2-e2f3cf9f18c1","Type":"ContainerStarted","Data":"a6c89eac32c250fb0717551c5fdc0142ba230f157fad1f9c114da3abae5df0da"}
Oct 31 13:12:25.063604 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:25.063543    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv" podStartSLOduration=1.063525436 podStartE2EDuration="1.063525436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:25.06120031 +0000 UTC m=+2669.801626539" watchObservedRunningTime="2026-10-31 13:12:25.063525436 +0000 UTC m=+2669.803951653"
Oct 31 13:12:26.049478 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:26.049443    5663 generic.go:356] "Generic (PLEG): container finished" podID="8a257c14-3432-4628-b2e2-e2f3cf9f18c1" containerID="2c50f408782ac1c3099f345103f6f646ee8548c44c56006f1387bfad0052464d" exitCode=0
Oct 31 13:12:26.049894 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:26.049502    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv" event={"ID":"8a257c14-3432-4628-b2e2-e2f3cf9f18c1","Type":"ContainerDied","Data":"2c50f408782ac1c3099f345103f6f646ee8548c44c56006f1387bfad0052464d"}
Oct 31 13:12:27.179959 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:27.179934    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:27.341773 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:27.341742    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-xks-ca\") pod \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\" (UID: \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\") "
Oct 31 13:12:27.341925 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:27.341816    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-openshift-ca\") pod \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\" (UID: \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\") "
Oct 31 13:12:27.341925 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:27.341847    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-kube-api-access-8khgk\" (UniqueName: \"kubernetes.io/projected/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-kube-api-access-8khgk\") pod \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\" (UID: \"8a257c14-3432-4628-b2e2-e2f3cf9f18c1\") "
Oct 31 13:12:27.342195 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:27.342169    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-openshift-ca" pod "8a257c14-3432-4628-b2e2-e2f3cf9f18c1" (UID: "8a257c14-3432-4628-b2e2-e2f3cf9f18c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:27.343877 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:27.343848    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-xks-ca" pod "8a257c14-3432-4628-b2e2-e2f3cf9f18c1" (UID: "8a257c14-3432-4628-b2e2-e2f3cf9f18c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:27.343973 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:27.343950    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-kube-api-access-8khgk" pod "8a257c14-3432-4628-b2e2-e2f3cf9f18c1" (UID: "8a257c14-3432-4628-b2e2-e2f3cf9f18c1"). InnerVolumeSpecName "kube-api-access-8khgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:27.442737 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:27.442707    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:27.442737 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:27.442731    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:27.442737 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:27.442742    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8khgk\" (UniqueName: \"kubernetes.io/projected/8a257c14-3432-4628-b2e2-e2f3cf9f18c1-kube-api-access-8khgk\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:28.060274 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:28.060237    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv" event={"ID":"8a257c14-3432-4628-b2e2-e2f3cf9f18c1","Type":"ContainerDied","Data":"a6c89eac32c250fb0717551c5fdc0142ba230f157fad1f9c114da3abae5df0da"}
Oct 31 13:12:28.060572 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:28.060281    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"
Oct 31 13:12:28.060572 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:28.060296    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6c89eac32c250fb0717551c5fdc0142ba230f157fad1f9c114da3abae5df0da"
Oct 31 13:12:33.304396 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.304308    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg" podUID="dd629212-3415-46ab-a6c9-ca7518a1925e" containerName="main" containerID="cri-o://7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460" gracePeriod=45
Oct 31 13:12:33.564055 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.564033    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:12:33.701137 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.701100    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd629212-3415-46ab-a6c9-ca7518a1925e-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd629212-3415-46ab-a6c9-ca7518a1925e-tls-certs\") pod \"dd629212-3415-46ab-a6c9-ca7518a1925e\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") "
Oct 31 13:12:33.701380 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.701154    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-model-cache\") pod \"dd629212-3415-46ab-a6c9-ca7518a1925e\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") "
Oct 31 13:12:33.701380 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.701211    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-tmp-dir\") pod \"dd629212-3415-46ab-a6c9-ca7518a1925e\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") "
Oct 31 13:12:33.701380 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.701240    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd629212-3415-46ab-a6c9-ca7518a1925e-kube-api-access-5x4z2\" (UniqueName: \"kubernetes.io/projected/dd629212-3415-46ab-a6c9-ca7518a1925e-kube-api-access-5x4z2\") pod \"dd629212-3415-46ab-a6c9-ca7518a1925e\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") "
Oct 31 13:12:33.701380 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.701265    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-home\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-home\") pod \"dd629212-3415-46ab-a6c9-ca7518a1925e\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") "
Oct 31 13:12:33.701380 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.701302    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-dshm\") pod \"dd629212-3415-46ab-a6c9-ca7518a1925e\" (UID: \"dd629212-3415-46ab-a6c9-ca7518a1925e\") "
Oct 31 13:12:33.701663 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.701525    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-tmp-dir" pod "dd629212-3415-46ab-a6c9-ca7518a1925e" (UID: "dd629212-3415-46ab-a6c9-ca7518a1925e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:12:33.701663 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.701543    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-model-cache" pod "dd629212-3415-46ab-a6c9-ca7518a1925e" (UID: "dd629212-3415-46ab-a6c9-ca7518a1925e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:12:33.701663 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.701615    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-home" pod "dd629212-3415-46ab-a6c9-ca7518a1925e" (UID: "dd629212-3415-46ab-a6c9-ca7518a1925e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:12:33.703476 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.703450    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-dshm" pod "dd629212-3415-46ab-a6c9-ca7518a1925e" (UID: "dd629212-3415-46ab-a6c9-ca7518a1925e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:12:33.703581 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.703505    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd629212-3415-46ab-a6c9-ca7518a1925e-tls-certs" pod "dd629212-3415-46ab-a6c9-ca7518a1925e" (UID: "dd629212-3415-46ab-a6c9-ca7518a1925e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:33.703632 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.703603    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd629212-3415-46ab-a6c9-ca7518a1925e-kube-api-access-5x4z2" pod "dd629212-3415-46ab-a6c9-ca7518a1925e" (UID: "dd629212-3415-46ab-a6c9-ca7518a1925e"). InnerVolumeSpecName "kube-api-access-5x4z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:33.802994 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.802954    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd629212-3415-46ab-a6c9-ca7518a1925e-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:33.802994 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.802987    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:33.803202 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.803001    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:33.803202 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.803016    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x4z2\" (UniqueName: \"kubernetes.io/projected/dd629212-3415-46ab-a6c9-ca7518a1925e-kube-api-access-5x4z2\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:33.803202 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.803029    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:33.803202 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:33.803040    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd629212-3415-46ab-a6c9-ca7518a1925e-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:34.088515 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:34.088483    5663 generic.go:356] "Generic (PLEG): container finished" podID="dd629212-3415-46ab-a6c9-ca7518a1925e" containerID="7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460" exitCode=0
Oct 31 13:12:34.088720 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:34.088529    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"
Oct 31 13:12:34.088720 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:34.088528    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg" event={"ID":"dd629212-3415-46ab-a6c9-ca7518a1925e","Type":"ContainerDied","Data":"7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460"}
Oct 31 13:12:34.088720 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:34.088654    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg" event={"ID":"dd629212-3415-46ab-a6c9-ca7518a1925e","Type":"ContainerDied","Data":"9e606c5f5069a141cfac044b3dd6aaa54367732624d7dd13a8134b8d9c778cd0"}
Oct 31 13:12:34.088720 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:34.088685    5663 scope.go:122] "RemoveContainer" containerID="7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460"
Oct 31 13:12:34.098188 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:34.098163    5663 scope.go:122] "RemoveContainer" containerID="7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460"
Oct 31 13:12:34.098542 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:12:34.098521    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460\": container with ID starting with 7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460 not found: ID does not exist" containerID="7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460"
Oct 31 13:12:34.098633 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:34.098550    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460"} err="failed to get container status \"7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460\": rpc error: code = NotFound desc = could not find container \"7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460\": container with ID starting with 7e869ebc38dd011a009381f7a02043754664a11ccab4e421c6d95642f1664460 not found: ID does not exist"
Oct 31 13:12:34.113060 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:34.113035    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"]
Oct 31 13:12:34.119281 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:34.119258    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-26740100-kserve-7b87755d9c-pwmmg"]
Oct 31 13:12:35.443713 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:35.443211    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd629212-3415-46ab-a6c9-ca7518a1925e" path="/var/lib/kubelet/pods/dd629212-3415-46ab-a6c9-ca7518a1925e/volumes"
Oct 31 13:12:35.587736 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:35.587693    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc" podUID="8e00fbee-619e-43c1-ab75-b468f1bae823" containerName="main" containerID="cri-o://1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174" gracePeriod=45
Oct 31 13:12:36.046486 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.046458    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:12:36.136376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.135505    5663 generic.go:356] "Generic (PLEG): container finished" podID="8e00fbee-619e-43c1-ab75-b468f1bae823" containerID="1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174" exitCode=0
Oct 31 13:12:36.136376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.135559    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc" event={"ID":"8e00fbee-619e-43c1-ab75-b468f1bae823","Type":"ContainerDied","Data":"1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174"}
Oct 31 13:12:36.136376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.135595    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc" event={"ID":"8e00fbee-619e-43c1-ab75-b468f1bae823","Type":"ContainerDied","Data":"fab79e726b3f1472c9b2e678c79c925caedc700f18da51df2a996d7051e19975"}
Oct 31 13:12:36.136376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.135632    5663 scope.go:122] "RemoveContainer" containerID="1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174"
Oct 31 13:12:36.136376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.135770    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"
Oct 31 13:12:36.148234 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.148215    5663 scope.go:122] "RemoveContainer" containerID="1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174"
Oct 31 13:12:36.148603 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:12:36.148567    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174\": container with ID starting with 1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174 not found: ID does not exist" containerID="1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174"
Oct 31 13:12:36.148703 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.148606    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174"} err="failed to get container status \"1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174\": rpc error: code = NotFound desc = could not find container \"1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174\": container with ID starting with 1b77b961c087b49e5b238d5efb074c27b29d1a48b0ddb8ca50c97364ec84f174 not found: ID does not exist"
Oct 31 13:12:36.242638 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.242597    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-model-cache\") pod \"8e00fbee-619e-43c1-ab75-b468f1bae823\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") "
Oct 31 13:12:36.242849 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.242687    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-dshm\") pod \"8e00fbee-619e-43c1-ab75-b468f1bae823\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") "
Oct 31 13:12:36.242849 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.242715    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-tmp-dir\") pod \"8e00fbee-619e-43c1-ab75-b468f1bae823\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") "
Oct 31 13:12:36.242849 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.242787    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-home\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-home\") pod \"8e00fbee-619e-43c1-ab75-b468f1bae823\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") "
Oct 31 13:12:36.242849 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.242820    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e00fbee-619e-43c1-ab75-b468f1bae823-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e00fbee-619e-43c1-ab75-b468f1bae823-tls-certs\") pod \"8e00fbee-619e-43c1-ab75-b468f1bae823\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") "
Oct 31 13:12:36.243068 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.242869    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e00fbee-619e-43c1-ab75-b468f1bae823-kube-api-access-qw8n2\" (UniqueName: \"kubernetes.io/projected/8e00fbee-619e-43c1-ab75-b468f1bae823-kube-api-access-qw8n2\") pod \"8e00fbee-619e-43c1-ab75-b468f1bae823\" (UID: \"8e00fbee-619e-43c1-ab75-b468f1bae823\") "
Oct 31 13:12:36.243068 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.243024    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-model-cache" pod "8e00fbee-619e-43c1-ab75-b468f1bae823" (UID: "8e00fbee-619e-43c1-ab75-b468f1bae823"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:12:36.243177 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.243165    5663 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-model-cache\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:36.243288 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.243262    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-home" pod "8e00fbee-619e-43c1-ab75-b468f1bae823" (UID: "8e00fbee-619e-43c1-ab75-b468f1bae823"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:12:36.243648 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.243602    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-tmp-dir" pod "8e00fbee-619e-43c1-ab75-b468f1bae823" (UID: "8e00fbee-619e-43c1-ab75-b468f1bae823"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:12:36.245509 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.245487    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e00fbee-619e-43c1-ab75-b468f1bae823-kube-api-access-qw8n2" pod "8e00fbee-619e-43c1-ab75-b468f1bae823" (UID: "8e00fbee-619e-43c1-ab75-b468f1bae823"). InnerVolumeSpecName "kube-api-access-qw8n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:36.245780 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.245759    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e00fbee-619e-43c1-ab75-b468f1bae823-tls-certs" pod "8e00fbee-619e-43c1-ab75-b468f1bae823" (UID: "8e00fbee-619e-43c1-ab75-b468f1bae823"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:36.245852 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.245767    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-dshm" pod "8e00fbee-619e-43c1-ab75-b468f1bae823" (UID: "8e00fbee-619e-43c1-ab75-b468f1bae823"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:12:36.343609 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.343570    5663 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-dshm\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:36.343787 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.343619    5663 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-tmp-dir\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:36.343787 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.343629    5663 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e00fbee-619e-43c1-ab75-b468f1bae823-home\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:36.343787 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.343639    5663 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e00fbee-619e-43c1-ab75-b468f1bae823-tls-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:36.343787 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.343649    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw8n2\" (UniqueName: \"kubernetes.io/projected/8e00fbee-619e-43c1-ab75-b468f1bae823-kube-api-access-qw8n2\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:36.464374 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.462820    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"]
Oct 31 13:12:36.482376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.481192    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc"]
Oct 31 13:12:36.835697 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:36.835572    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-26740100-kserve-79c4d97969-7b7vc" podUID="8e00fbee-619e-43c1-ab75-b468f1bae823" containerName="main" probeResult="failure" output="Get \"https://10.133.0.178:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:12:37.438722 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:37.438684    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e00fbee-619e-43c1-ab75-b468f1bae823" path="/var/lib/kubelet/pods/8e00fbee-619e-43c1-ab75-b468f1bae823/volumes"
Oct 31 13:12:45.296511 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.296475    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"]
Oct 31 13:12:45.296928 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.296840    5663 cpu_manager.go:414] "containerMap: removing container" podUID="8a257c14-3432-4628-b2e2-e2f3cf9f18c1" containerName="revoke-keys"
Oct 31 13:12:45.296928 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.296854    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a257c14-3432-4628-b2e2-e2f3cf9f18c1" containerName="revoke-keys"
Oct 31 13:12:45.296928 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.296877    5663 cpu_manager.go:414] "containerMap: removing container" podUID="dd629212-3415-46ab-a6c9-ca7518a1925e" containerName="main"
Oct 31 13:12:45.296928 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.296883    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd629212-3415-46ab-a6c9-ca7518a1925e" containerName="main"
Oct 31 13:12:45.296928 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.296898    5663 cpu_manager.go:414] "containerMap: removing container" podUID="8e00fbee-619e-43c1-ab75-b468f1bae823" containerName="main"
Oct 31 13:12:45.296928 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.296903    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e00fbee-619e-43c1-ab75-b468f1bae823" containerName="main"
Oct 31 13:12:45.297130 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.296952    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e00fbee-619e-43c1-ab75-b468f1bae823" containerName="main"
Oct 31 13:12:45.297130 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.296963    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a257c14-3432-4628-b2e2-e2f3cf9f18c1" containerName="revoke-keys"
Oct 31 13:12:45.297130 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.296972    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd629212-3415-46ab-a6c9-ca7518a1925e" containerName="main"
Oct 31 13:12:45.301406 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.301387    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:45.308064 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.308038    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:12:45.308246 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.308106    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wsmb\""
Oct 31 13:12:45.310060 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.310034    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"]
Oct 31 13:12:45.365824 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.365788    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f7bebc-7td2r\" (UID: \"48a3d515-18f3-4084-8b25-60ca30514d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:45.365824 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.365819    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpdxl\" (UniqueName: \"kubernetes.io/projected/48a3d515-18f3-4084-8b25-60ca30514d21-kube-api-access-gpdxl\") pod \"maas-api-revoke-sub-e4f7bebc-7td2r\" (UID: \"48a3d515-18f3-4084-8b25-60ca30514d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:45.366040 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.365853    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f7bebc-7td2r\" (UID: \"48a3d515-18f3-4084-8b25-60ca30514d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:45.466417 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.466347    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f7bebc-7td2r\" (UID: \"48a3d515-18f3-4084-8b25-60ca30514d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:45.466417 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.466422    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpdxl\" (UniqueName: \"kubernetes.io/projected/48a3d515-18f3-4084-8b25-60ca30514d21-kube-api-access-gpdxl\") pod \"maas-api-revoke-sub-e4f7bebc-7td2r\" (UID: \"48a3d515-18f3-4084-8b25-60ca30514d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:45.466649 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.466475    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f7bebc-7td2r\" (UID: \"48a3d515-18f3-4084-8b25-60ca30514d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:45.467046 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.467026    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f7bebc-7td2r\" (UID: \"48a3d515-18f3-4084-8b25-60ca30514d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:45.468849 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.468827    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f7bebc-7td2r\" (UID: \"48a3d515-18f3-4084-8b25-60ca30514d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:45.478775 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.478746    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpdxl\" (UniqueName: \"kubernetes.io/projected/48a3d515-18f3-4084-8b25-60ca30514d21-kube-api-access-gpdxl\") pod \"maas-api-revoke-sub-e4f7bebc-7td2r\" (UID: \"48a3d515-18f3-4084-8b25-60ca30514d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:45.615896 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.615849    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:45.888278 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:45.888108    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"]
Oct 31 13:12:45.891039 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:12:45.891011    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a3d515_18f3_4084_8b25_60ca30514d21.slice/crio-f2484dcf34e8231e50d83527ccaacae30811977e8dbe09bd164c0dca5c5c215f WatchSource:0}: Error finding container f2484dcf34e8231e50d83527ccaacae30811977e8dbe09bd164c0dca5c5c215f: Status 404 returned error can't find the container with id f2484dcf34e8231e50d83527ccaacae30811977e8dbe09bd164c0dca5c5c215f
Oct 31 13:12:46.203706 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:46.203608    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r" event={"ID":"48a3d515-18f3-4084-8b25-60ca30514d21","Type":"ContainerStarted","Data":"986785738c79e65a46fc95b45a2913bccb92e0485717067f9eee30ec2cdfbc2b"}
Oct 31 13:12:46.203706 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:46.203669    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r" event={"ID":"48a3d515-18f3-4084-8b25-60ca30514d21","Type":"ContainerStarted","Data":"f2484dcf34e8231e50d83527ccaacae30811977e8dbe09bd164c0dca5c5c215f"}
Oct 31 13:12:46.221789 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:46.221734    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r" podStartSLOduration=1.221720721 podStartE2EDuration="1.221720721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:46.220566415 +0000 UTC m=+2690.960992632" watchObservedRunningTime="2026-10-31 13:12:46.221720721 +0000 UTC m=+2690.962146937"
Oct 31 13:12:47.209515 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:47.209482    5663 generic.go:356] "Generic (PLEG): container finished" podID="48a3d515-18f3-4084-8b25-60ca30514d21" containerID="986785738c79e65a46fc95b45a2913bccb92e0485717067f9eee30ec2cdfbc2b" exitCode=0
Oct 31 13:12:47.209895 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:47.209571    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r" event={"ID":"48a3d515-18f3-4084-8b25-60ca30514d21","Type":"ContainerDied","Data":"986785738c79e65a46fc95b45a2913bccb92e0485717067f9eee30ec2cdfbc2b"}
Oct 31 13:12:48.339247 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:48.339218    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:48.395733 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:48.395699    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a3d515-18f3-4084-8b25-60ca30514d21-kube-api-access-gpdxl\" (UniqueName: \"kubernetes.io/projected/48a3d515-18f3-4084-8b25-60ca30514d21-kube-api-access-gpdxl\") pod \"48a3d515-18f3-4084-8b25-60ca30514d21\" (UID: \"48a3d515-18f3-4084-8b25-60ca30514d21\") "
Oct 31 13:12:48.395933 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:48.395758    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-xks-ca\") pod \"48a3d515-18f3-4084-8b25-60ca30514d21\" (UID: \"48a3d515-18f3-4084-8b25-60ca30514d21\") "
Oct 31 13:12:48.395933 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:48.395820    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-openshift-ca\") pod \"48a3d515-18f3-4084-8b25-60ca30514d21\" (UID: \"48a3d515-18f3-4084-8b25-60ca30514d21\") "
Oct 31 13:12:48.396124 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:48.396105    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-openshift-ca" pod "48a3d515-18f3-4084-8b25-60ca30514d21" (UID: "48a3d515-18f3-4084-8b25-60ca30514d21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:48.397956 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:48.397931    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a3d515-18f3-4084-8b25-60ca30514d21-kube-api-access-gpdxl" pod "48a3d515-18f3-4084-8b25-60ca30514d21" (UID: "48a3d515-18f3-4084-8b25-60ca30514d21"). InnerVolumeSpecName "kube-api-access-gpdxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:48.398027 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:48.397944    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-xks-ca" pod "48a3d515-18f3-4084-8b25-60ca30514d21" (UID: "48a3d515-18f3-4084-8b25-60ca30514d21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:48.497168 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:48.497085    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpdxl\" (UniqueName: \"kubernetes.io/projected/48a3d515-18f3-4084-8b25-60ca30514d21-kube-api-access-gpdxl\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:48.497168 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:48.497118    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-xks-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:48.497168 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:48.497131    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a3d515-18f3-4084-8b25-60ca30514d21-maas-api-openshift-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:49.218891 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:49.218851    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r" event={"ID":"48a3d515-18f3-4084-8b25-60ca30514d21","Type":"ContainerDied","Data":"f2484dcf34e8231e50d83527ccaacae30811977e8dbe09bd164c0dca5c5c215f"}
Oct 31 13:12:49.218891 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:49.218878    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"
Oct 31 13:12:49.219121 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:49.218925    5663 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2484dcf34e8231e50d83527ccaacae30811977e8dbe09bd164c0dca5c5c215f"
Oct 31 13:12:52.474883 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:12:52.474842    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:12:57.023165 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.023127    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"]
Oct 31 13:12:57.026282 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.026251    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96qkg"]
Oct 31 13:12:57.424787 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.424753    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"]
Oct 31 13:12:57.425131 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.425118    5663 cpu_manager.go:414] "containerMap: removing container" podUID="48a3d515-18f3-4084-8b25-60ca30514d21" containerName="revoke-keys"
Oct 31 13:12:57.425197 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.425132    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a3d515-18f3-4084-8b25-60ca30514d21" containerName="revoke-keys"
Oct 31 13:12:57.425197 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.425192    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a3d515-18f3-4084-8b25-60ca30514d21" containerName="revoke-keys"
Oct 31 13:12:57.428157 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.428140    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.439840 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.439812    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 13:12:57.439988 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.439926    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 13:12:57.439988 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.439978    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b4dc9d-761c-4ad1-9087-3e697cf17416" path="/var/lib/kubelet/pods/b1b4dc9d-761c-4ad1-9087-3e697cf17416/volumes"
Oct 31 13:12:57.440101 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.439998    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6jkzx\""
Oct 31 13:12:57.443868 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.443845    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"]
Oct 31 13:12:57.446708 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.446684    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"]
Oct 31 13:12:57.450636 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.450611    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.464580 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.464551    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"]
Oct 31 13:12:57.483123 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.483094    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a607dbc8-7f65-491d-9d5c-bb05e29dc431-openshift-service-ca\") pod \"maas-api-598767f7b6-qqvkg\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.483310 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.483151    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7x6c\" (UniqueName: \"kubernetes.io/projected/a607dbc8-7f65-491d-9d5c-bb05e29dc431-kube-api-access-z7x6c\") pod \"maas-api-598767f7b6-qqvkg\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.483310 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.483293    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-metrics-certs\") pod \"maas-api-598767f7b6-qqvkg\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.483442 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.483321    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-maas-api-tls\") pod \"maas-api-598767f7b6-qqvkg\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.583906 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.583874    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4st\" (UniqueName: \"kubernetes.io/projected/4814e92c-745a-480b-8b21-540d70d85cf6-kube-api-access-tc4st\") pod \"maas-api-7c54444cdb-4pldr\" (UID: \"4814e92c-745a-480b-8b21-540d70d85cf6\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.584099 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.583914    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4814e92c-745a-480b-8b21-540d70d85cf6-maas-api-tls\") pod \"maas-api-7c54444cdb-4pldr\" (UID: \"4814e92c-745a-480b-8b21-540d70d85cf6\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.584099 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.583943    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7x6c\" (UniqueName: \"kubernetes.io/projected/a607dbc8-7f65-491d-9d5c-bb05e29dc431-kube-api-access-z7x6c\") pod \"maas-api-598767f7b6-qqvkg\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.584099 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.584073    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-metrics-certs\") pod \"maas-api-598767f7b6-qqvkg\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.584283 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.584114    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4814e92c-745a-480b-8b21-540d70d85cf6-openshift-service-ca\") pod \"maas-api-7c54444cdb-4pldr\" (UID: \"4814e92c-745a-480b-8b21-540d70d85cf6\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.584283 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.584149    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-maas-api-tls\") pod \"maas-api-598767f7b6-qqvkg\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.584283 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.584226    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4814e92c-745a-480b-8b21-540d70d85cf6-metrics-certs\") pod \"maas-api-7c54444cdb-4pldr\" (UID: \"4814e92c-745a-480b-8b21-540d70d85cf6\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.584283 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.584262    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a607dbc8-7f65-491d-9d5c-bb05e29dc431-openshift-service-ca\") pod \"maas-api-598767f7b6-qqvkg\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.584884 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.584867    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a607dbc8-7f65-491d-9d5c-bb05e29dc431-openshift-service-ca\") pod \"maas-api-598767f7b6-qqvkg\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.586680 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.586653    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-metrics-certs\") pod \"maas-api-598767f7b6-qqvkg\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.586825 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.586808    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-maas-api-tls\") pod \"maas-api-598767f7b6-qqvkg\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.596204 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.596180    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7x6c\" (UniqueName: \"kubernetes.io/projected/a607dbc8-7f65-491d-9d5c-bb05e29dc431-kube-api-access-z7x6c\") pod \"maas-api-598767f7b6-qqvkg\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") " pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.685335 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.685243    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4st\" (UniqueName: \"kubernetes.io/projected/4814e92c-745a-480b-8b21-540d70d85cf6-kube-api-access-tc4st\") pod \"maas-api-7c54444cdb-4pldr\" (UID: \"4814e92c-745a-480b-8b21-540d70d85cf6\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.685335 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.685286    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4814e92c-745a-480b-8b21-540d70d85cf6-maas-api-tls\") pod \"maas-api-7c54444cdb-4pldr\" (UID: \"4814e92c-745a-480b-8b21-540d70d85cf6\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.685553 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.685342    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4814e92c-745a-480b-8b21-540d70d85cf6-openshift-service-ca\") pod \"maas-api-7c54444cdb-4pldr\" (UID: \"4814e92c-745a-480b-8b21-540d70d85cf6\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.685553 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.685384    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4814e92c-745a-480b-8b21-540d70d85cf6-metrics-certs\") pod \"maas-api-7c54444cdb-4pldr\" (UID: \"4814e92c-745a-480b-8b21-540d70d85cf6\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.686110 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.686084    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4814e92c-745a-480b-8b21-540d70d85cf6-openshift-service-ca\") pod \"maas-api-7c54444cdb-4pldr\" (UID: \"4814e92c-745a-480b-8b21-540d70d85cf6\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.687759 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.687739    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4814e92c-745a-480b-8b21-540d70d85cf6-maas-api-tls\") pod \"maas-api-7c54444cdb-4pldr\" (UID: \"4814e92c-745a-480b-8b21-540d70d85cf6\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.687963 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.687940    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4814e92c-745a-480b-8b21-540d70d85cf6-metrics-certs\") pod \"maas-api-7c54444cdb-4pldr\" (UID: \"4814e92c-745a-480b-8b21-540d70d85cf6\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.699762 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.699732    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4st\" (UniqueName: \"kubernetes.io/projected/4814e92c-745a-480b-8b21-540d70d85cf6-kube-api-access-tc4st\") pod \"maas-api-7c54444cdb-4pldr\" (UID: \"4814e92c-745a-480b-8b21-540d70d85cf6\") " pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:57.743312 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.743279    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:57.765600 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:57.765569    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:58.022908 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:58.022768    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"]
Oct 31 13:12:58.025947 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:12:58.025912    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda607dbc8_7f65_491d_9d5c_bb05e29dc431.slice/crio-b8e533d4395a98a4708510d888b4a486e527073df146ea3366e185c32c4c1710 WatchSource:0}: Error finding container b8e533d4395a98a4708510d888b4a486e527073df146ea3366e185c32c4c1710: Status 404 returned error can't find the container with id b8e533d4395a98a4708510d888b4a486e527073df146ea3366e185c32c4c1710
Oct 31 13:12:58.027401 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:58.027383    5663 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:12:58.049382 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:58.049341    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"]
Oct 31 13:12:58.051936 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:12:58.051909    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4814e92c_745a_480b_8b21_540d70d85cf6.slice/crio-fa65352a3f0e1539aa650f0184c76dda25c90b49e79aa5bc175335f8c1411b34 WatchSource:0}: Error finding container fa65352a3f0e1539aa650f0184c76dda25c90b49e79aa5bc175335f8c1411b34: Status 404 returned error can't find the container with id fa65352a3f0e1539aa650f0184c76dda25c90b49e79aa5bc175335f8c1411b34
Oct 31 13:12:58.262364 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:58.262312    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg" event={"ID":"a607dbc8-7f65-491d-9d5c-bb05e29dc431","Type":"ContainerStarted","Data":"b8e533d4395a98a4708510d888b4a486e527073df146ea3366e185c32c4c1710"}
Oct 31 13:12:58.263466 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:58.263438    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr" event={"ID":"4814e92c-745a-480b-8b21-540d70d85cf6","Type":"ContainerStarted","Data":"fa65352a3f0e1539aa650f0184c76dda25c90b49e79aa5bc175335f8c1411b34"}
Oct 31 13:12:59.269559 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:59.269513    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg" event={"ID":"a607dbc8-7f65-491d-9d5c-bb05e29dc431","Type":"ContainerStarted","Data":"acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22"}
Oct 31 13:12:59.270153 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:59.269651    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:12:59.270973 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:59.270948    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr" event={"ID":"4814e92c-745a-480b-8b21-540d70d85cf6","Type":"ContainerStarted","Data":"c182572940dece368869e8e178da66a3b1c662456b3d3d35c79b9f0e32ae3c58"}
Oct 31 13:12:59.271085 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:59.271064    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:12:59.290190 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:59.290142    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg" podStartSLOduration=1.784476498 podStartE2EDuration="2.290128445s" totalImagesPullingTime="505.651947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:59.289439239 +0000 UTC m=+2704.029865459" watchObservedRunningTime="2026-10-31 13:12:59.290128445 +0000 UTC m=+2704.030554695"
Oct 31 13:12:59.310530 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:59.310479    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr" podStartSLOduration=1.859572171 podStartE2EDuration="2.310464457s" totalImagesPullingTime="450.892286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:59.309115327 +0000 UTC m=+2704.049541543" watchObservedRunningTime="2026-10-31 13:12:59.310464457 +0000 UTC m=+2704.050890674"
Oct 31 13:12:59.789876 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:59.789847    5663 scope.go:122] "RemoveContainer" containerID="2cd488a3b8c13a5de3f9edb22d7da1c5ae12a2cdf1c33cf7e6c7f97fb3fff665"
Oct 31 13:12:59.798083 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:59.798058    5663 scope.go:122] "RemoveContainer" containerID="a9bcfecdd2f91ef16954a5b7924889dd26bd2decd55e0a869a4f9a04c53f94b7"
Oct 31 13:12:59.806530 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:12:59.806511    5663 scope.go:122] "RemoveContainer" containerID="3970cff077d4d2c88a74f42cba5fce3db543b87703f653757290dcf206f8a38f"
Oct 31 13:13:05.281545 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.281321    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c54444cdb-4pldr"
Oct 31 13:13:05.281950 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.281783    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:13:05.357474 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.357440    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"]
Oct 31 13:13:05.357693 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.357647    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg" podUID="a607dbc8-7f65-491d-9d5c-bb05e29dc431" containerName="maas-api" containerID="cri-o://acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22" gracePeriod=30
Oct 31 13:13:05.609895 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.609870    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:13:05.757181 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.757149    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a607dbc8-7f65-491d-9d5c-bb05e29dc431-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a607dbc8-7f65-491d-9d5c-bb05e29dc431-openshift-service-ca\") pod \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") "
Oct 31 13:13:05.757412 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.757208    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-metrics-certs\") pod \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") "
Oct 31 13:13:05.757412 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.757308    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a607dbc8-7f65-491d-9d5c-bb05e29dc431-kube-api-access-z7x6c\" (UniqueName: \"kubernetes.io/projected/a607dbc8-7f65-491d-9d5c-bb05e29dc431-kube-api-access-z7x6c\") pod \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") "
Oct 31 13:13:05.757412 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.757376    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-maas-api-tls\") pod \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\" (UID: \"a607dbc8-7f65-491d-9d5c-bb05e29dc431\") "
Oct 31 13:13:05.757644 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.757618    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a607dbc8-7f65-491d-9d5c-bb05e29dc431-openshift-service-ca" pod "a607dbc8-7f65-491d-9d5c-bb05e29dc431" (UID: "a607dbc8-7f65-491d-9d5c-bb05e29dc431"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:13:05.759463 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.759436    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-metrics-certs" pod "a607dbc8-7f65-491d-9d5c-bb05e29dc431" (UID: "a607dbc8-7f65-491d-9d5c-bb05e29dc431"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:13:05.759463 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.759454    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-maas-api-tls" pod "a607dbc8-7f65-491d-9d5c-bb05e29dc431" (UID: "a607dbc8-7f65-491d-9d5c-bb05e29dc431"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:13:05.759620 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.759458    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a607dbc8-7f65-491d-9d5c-bb05e29dc431-kube-api-access-z7x6c" pod "a607dbc8-7f65-491d-9d5c-bb05e29dc431" (UID: "a607dbc8-7f65-491d-9d5c-bb05e29dc431"). InnerVolumeSpecName "kube-api-access-z7x6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:13:05.859142 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.859057    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7x6c\" (UniqueName: \"kubernetes.io/projected/a607dbc8-7f65-491d-9d5c-bb05e29dc431-kube-api-access-z7x6c\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:05.859142 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.859088    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:05.859142 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.859098    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a607dbc8-7f65-491d-9d5c-bb05e29dc431-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:05.859142 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:05.859106    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a607dbc8-7f65-491d-9d5c-bb05e29dc431-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:06.304135 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:06.304039    5663 generic.go:356] "Generic (PLEG): container finished" podID="a607dbc8-7f65-491d-9d5c-bb05e29dc431" containerID="acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22" exitCode=0
Oct 31 13:13:06.304135 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:06.304082    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"
Oct 31 13:13:06.304637 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:06.304134    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg" event={"ID":"a607dbc8-7f65-491d-9d5c-bb05e29dc431","Type":"ContainerDied","Data":"acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22"}
Oct 31 13:13:06.304637 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:06.304190    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg" event={"ID":"a607dbc8-7f65-491d-9d5c-bb05e29dc431","Type":"ContainerDied","Data":"b8e533d4395a98a4708510d888b4a486e527073df146ea3366e185c32c4c1710"}
Oct 31 13:13:06.304637 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:06.304213    5663 scope.go:122] "RemoveContainer" containerID="acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22"
Oct 31 13:13:06.313579 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:06.313561    5663 scope.go:122] "RemoveContainer" containerID="acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22"
Oct 31 13:13:06.313843 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:13:06.313824    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22\": container with ID starting with acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22 not found: ID does not exist" containerID="acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22"
Oct 31 13:13:06.313894 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:06.313853    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22"} err="failed to get container status \"acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22\": rpc error: code = NotFound desc = could not find container \"acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22\": container with ID starting with acc28573b0fdc15f006d9edc0eb2dc85629dfb387520e232745f70679bd91f22 not found: ID does not exist"
Oct 31 13:13:06.328604 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:06.328581    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"]
Oct 31 13:13:06.333000 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:06.332982    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-598767f7b6-qqvkg"]
Oct 31 13:13:07.437032 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:07.436993    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a607dbc8-7f65-491d-9d5c-bb05e29dc431" path="/var/lib/kubelet/pods/a607dbc8-7f65-491d-9d5c-bb05e29dc431/volumes"
Oct 31 13:13:08.973776 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:08.973741    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"]
Oct 31 13:13:08.974249 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:08.974150    5663 cpu_manager.go:414] "containerMap: removing container" podUID="a607dbc8-7f65-491d-9d5c-bb05e29dc431" containerName="maas-api"
Oct 31 13:13:08.974249 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:08.974168    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a607dbc8-7f65-491d-9d5c-bb05e29dc431" containerName="maas-api"
Oct 31 13:13:08.974335 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:08.974250    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="a607dbc8-7f65-491d-9d5c-bb05e29dc431" containerName="maas-api"
Oct 31 13:13:08.978588 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:08.978569    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"
Oct 31 13:13:08.985971 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:08.985949    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bbf78971\""
Oct 31 13:13:08.996858 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:08.996830    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"]
Oct 31 13:13:09.008364 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.008325    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"]
Oct 31 13:13:09.011622 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.011602    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"
Oct 31 13:13:09.021536 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.021512    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bbf78971-dockercfg-xqxxg\""
Oct 31 13:13:09.023169 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.023144    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"]
Oct 31 13:13:09.057793 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.057763    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"]
Oct 31 13:13:09.061267 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.061244    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.072778 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.072739    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"]
Oct 31 13:13:09.094590 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.094555    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bhpx\" (UniqueName: \"kubernetes.io/projected/808b8f1a-2d96-4b02-964e-1558e20bd57d-kube-api-access-9bhpx\") pod \"payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl\" (UID: \"808b8f1a-2d96-4b02-964e-1558e20bd57d\") " pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"
Oct 31 13:13:09.094784 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.094621    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/808b8f1a-2d96-4b02-964e-1558e20bd57d-plugins-config-volume\") pod \"payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl\" (UID: \"808b8f1a-2d96-4b02-964e-1558e20bd57d\") " pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"
Oct 31 13:13:09.121246 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.121218    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bbf78971-serving-cert\""
Oct 31 13:13:09.195981 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.195944    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjx7q\" (UniqueName: \"kubernetes.io/projected/b7032ceb-75ec-4f69-b174-338e0b964f07-kube-api-access-fjx7q\") pod \"maas-api-e2e-ait-bbf78971-dddf54d77-24lzw\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.196298 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.196272    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/808b8f1a-2d96-4b02-964e-1558e20bd57d-plugins-config-volume\") pod \"payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl\" (UID: \"808b8f1a-2d96-4b02-964e-1558e20bd57d\") " pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"
Oct 31 13:13:09.196439 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.196333    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6959288f-5224-4d72-86a9-3063ea6a59af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv\" (UID: \"6959288f-5224-4d72-86a9-3063ea6a59af\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"
Oct 31 13:13:09.196439 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.196407    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7032ceb-75ec-4f69-b174-338e0b964f07-openshift-service-ca\") pod \"maas-api-e2e-ait-bbf78971-dddf54d77-24lzw\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.196589 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.196459    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x6zw\" (UniqueName: \"kubernetes.io/projected/6959288f-5224-4d72-86a9-3063ea6a59af-kube-api-access-6x6zw\") pod \"payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv\" (UID: \"6959288f-5224-4d72-86a9-3063ea6a59af\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"
Oct 31 13:13:09.196644 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.196615    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-metrics-certs\") pod \"maas-api-e2e-ait-bbf78971-dddf54d77-24lzw\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.196727 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.196711    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-maas-api-tls\") pod \"maas-api-e2e-ait-bbf78971-dddf54d77-24lzw\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.196928 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.196902    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bhpx\" (UniqueName: \"kubernetes.io/projected/808b8f1a-2d96-4b02-964e-1558e20bd57d-kube-api-access-9bhpx\") pod \"payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl\" (UID: \"808b8f1a-2d96-4b02-964e-1558e20bd57d\") " pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"
Oct 31 13:13:09.197379 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.197336    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/808b8f1a-2d96-4b02-964e-1558e20bd57d-plugins-config-volume\") pod \"payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl\" (UID: \"808b8f1a-2d96-4b02-964e-1558e20bd57d\") " pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"
Oct 31 13:13:09.209877 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.209850    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bhpx\" (UniqueName: \"kubernetes.io/projected/808b8f1a-2d96-4b02-964e-1558e20bd57d-kube-api-access-9bhpx\") pod \"payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl\" (UID: \"808b8f1a-2d96-4b02-964e-1558e20bd57d\") " pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"
Oct 31 13:13:09.292087 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.292008    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"
Oct 31 13:13:09.298551 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.298521    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6959288f-5224-4d72-86a9-3063ea6a59af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv\" (UID: \"6959288f-5224-4d72-86a9-3063ea6a59af\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"
Oct 31 13:13:09.298680 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.298585    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7032ceb-75ec-4f69-b174-338e0b964f07-openshift-service-ca\") pod \"maas-api-e2e-ait-bbf78971-dddf54d77-24lzw\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.298680 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.298638    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x6zw\" (UniqueName: \"kubernetes.io/projected/6959288f-5224-4d72-86a9-3063ea6a59af-kube-api-access-6x6zw\") pod \"payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv\" (UID: \"6959288f-5224-4d72-86a9-3063ea6a59af\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"
Oct 31 13:13:09.298770 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.298675    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-metrics-certs\") pod \"maas-api-e2e-ait-bbf78971-dddf54d77-24lzw\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.298770 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.298721    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-maas-api-tls\") pod \"maas-api-e2e-ait-bbf78971-dddf54d77-24lzw\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.298848 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.298795    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjx7q\" (UniqueName: \"kubernetes.io/projected/b7032ceb-75ec-4f69-b174-338e0b964f07-kube-api-access-fjx7q\") pod \"maas-api-e2e-ait-bbf78971-dddf54d77-24lzw\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.299401 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.299333    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7032ceb-75ec-4f69-b174-338e0b964f07-openshift-service-ca\") pod \"maas-api-e2e-ait-bbf78971-dddf54d77-24lzw\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.299526 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.299443    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6959288f-5224-4d72-86a9-3063ea6a59af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv\" (UID: \"6959288f-5224-4d72-86a9-3063ea6a59af\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"
Oct 31 13:13:09.301706 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.301683    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-maas-api-tls\") pod \"maas-api-e2e-ait-bbf78971-dddf54d77-24lzw\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.301823 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.301688    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-metrics-certs\") pod \"maas-api-e2e-ait-bbf78971-dddf54d77-24lzw\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.321481 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.321447    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjx7q\" (UniqueName: \"kubernetes.io/projected/b7032ceb-75ec-4f69-b174-338e0b964f07-kube-api-access-fjx7q\") pod \"maas-api-e2e-ait-bbf78971-dddf54d77-24lzw\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.325099 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.325066    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x6zw\" (UniqueName: \"kubernetes.io/projected/6959288f-5224-4d72-86a9-3063ea6a59af-kube-api-access-6x6zw\") pod \"payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv\" (UID: \"6959288f-5224-4d72-86a9-3063ea6a59af\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"
Oct 31 13:13:09.325793 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.325732    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"
Oct 31 13:13:09.377742 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.377714    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:09.595989 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.595949    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"]
Oct 31 13:13:09.601545 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:13:09.601511    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808b8f1a_2d96_4b02_964e_1558e20bd57d.slice/crio-00717e5b5ff873579f95acde0265925f0f6b743a222bd9ae3cb5718e440131a1 WatchSource:0}: Error finding container 00717e5b5ff873579f95acde0265925f0f6b743a222bd9ae3cb5718e440131a1: Status 404 returned error can't find the container with id 00717e5b5ff873579f95acde0265925f0f6b743a222bd9ae3cb5718e440131a1
Oct 31 13:13:09.616859 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.616837    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"]
Oct 31 13:13:09.618848 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:13:09.618817    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6959288f_5224_4d72_86a9_3063ea6a59af.slice/crio-334dea71f74b670750e33eb5e5899c4324e4bfc4950a16311df7dd3e6593dd68 WatchSource:0}: Error finding container 334dea71f74b670750e33eb5e5899c4324e4bfc4950a16311df7dd3e6593dd68: Status 404 returned error can't find the container with id 334dea71f74b670750e33eb5e5899c4324e4bfc4950a16311df7dd3e6593dd68
Oct 31 13:13:09.673522 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:09.673496    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"]
Oct 31 13:13:09.675827 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:13:09.675803    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7032ceb_75ec_4f69_b174_338e0b964f07.slice/crio-8f3ea37dc4568962ddfe48776e9f207a67628527a3123c25e6fdf3213b3289cf WatchSource:0}: Error finding container 8f3ea37dc4568962ddfe48776e9f207a67628527a3123c25e6fdf3213b3289cf: Status 404 returned error can't find the container with id 8f3ea37dc4568962ddfe48776e9f207a67628527a3123c25e6fdf3213b3289cf
Oct 31 13:13:10.345422 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:10.345322    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv" event={"ID":"6959288f-5224-4d72-86a9-3063ea6a59af","Type":"ContainerStarted","Data":"40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4"}
Oct 31 13:13:10.345422 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:10.345403    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv" event={"ID":"6959288f-5224-4d72-86a9-3063ea6a59af","Type":"ContainerStarted","Data":"334dea71f74b670750e33eb5e5899c4324e4bfc4950a16311df7dd3e6593dd68"}
Oct 31 13:13:10.345911 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:10.345441    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"
Oct 31 13:13:10.346977 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:10.346946    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl" event={"ID":"808b8f1a-2d96-4b02-964e-1558e20bd57d","Type":"ContainerStarted","Data":"249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a"}
Oct 31 13:13:10.347128 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:10.346996    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl" event={"ID":"808b8f1a-2d96-4b02-964e-1558e20bd57d","Type":"ContainerStarted","Data":"00717e5b5ff873579f95acde0265925f0f6b743a222bd9ae3cb5718e440131a1"}
Oct 31 13:13:10.347128 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:10.347043    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"
Oct 31 13:13:10.348418 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:10.348390    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw" event={"ID":"b7032ceb-75ec-4f69-b174-338e0b964f07","Type":"ContainerStarted","Data":"d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a"}
Oct 31 13:13:10.348561 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:10.348431    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:10.348561 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:10.348447    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw" event={"ID":"b7032ceb-75ec-4f69-b174-338e0b964f07","Type":"ContainerStarted","Data":"8f3ea37dc4568962ddfe48776e9f207a67628527a3123c25e6fdf3213b3289cf"}
Oct 31 13:13:10.363073 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:10.363027    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv" podStartSLOduration=2.36301201 podStartE2EDuration="2.36301201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:10.362692125 +0000 UTC m=+2715.103118340" watchObservedRunningTime="2026-10-31 13:13:10.36301201 +0000 UTC m=+2715.103438227"
Oct 31 13:13:10.381677 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:10.381627    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl" podStartSLOduration=2.381614201 podStartE2EDuration="2.381614201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:10.381482938 +0000 UTC m=+2715.121909154" watchObservedRunningTime="2026-10-31 13:13:10.381614201 +0000 UTC m=+2715.122040417"
Oct 31 13:13:10.402023 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:10.401978    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw" podStartSLOduration=1.011736537 podStartE2EDuration="1.401967335s" totalImagesPullingTime="390.230798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:10.401021756 +0000 UTC m=+2715.141447972" watchObservedRunningTime="2026-10-31 13:13:10.401967335 +0000 UTC m=+2715.142393550"
Oct 31 13:13:16.358824 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:16.358795    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:21.355700 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:21.355671    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"
Oct 31 13:13:21.356255 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:21.355748    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"
Oct 31 13:13:22.475789 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:13:22.475754    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:13:33.463150 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:33.463114    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"]
Oct 31 13:13:33.480306 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:33.480275    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"]
Oct 31 13:13:33.796234 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:33.796150    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"]
Oct 31 13:13:33.796430 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:33.796348    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw" podUID="b7032ceb-75ec-4f69-b174-338e0b964f07" containerName="maas-api" containerID="cri-o://d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a" gracePeriod=30
Oct 31 13:13:34.058630 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.058606    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:34.127863 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.127824    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7032ceb-75ec-4f69-b174-338e0b964f07-kube-api-access-fjx7q\" (UniqueName: \"kubernetes.io/projected/b7032ceb-75ec-4f69-b174-338e0b964f07-kube-api-access-fjx7q\") pod \"b7032ceb-75ec-4f69-b174-338e0b964f07\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") "
Oct 31 13:13:34.128086 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.127910    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-metrics-certs\") pod \"b7032ceb-75ec-4f69-b174-338e0b964f07\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") "
Oct 31 13:13:34.128086 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.128067    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7032ceb-75ec-4f69-b174-338e0b964f07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7032ceb-75ec-4f69-b174-338e0b964f07-openshift-service-ca\") pod \"b7032ceb-75ec-4f69-b174-338e0b964f07\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") "
Oct 31 13:13:34.128228 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.128118    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-maas-api-tls\") pod \"b7032ceb-75ec-4f69-b174-338e0b964f07\" (UID: \"b7032ceb-75ec-4f69-b174-338e0b964f07\") "
Oct 31 13:13:34.128525 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.128487    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7032ceb-75ec-4f69-b174-338e0b964f07-openshift-service-ca" pod "b7032ceb-75ec-4f69-b174-338e0b964f07" (UID: "b7032ceb-75ec-4f69-b174-338e0b964f07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:13:34.130086 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.130055    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7032ceb-75ec-4f69-b174-338e0b964f07-kube-api-access-fjx7q" pod "b7032ceb-75ec-4f69-b174-338e0b964f07" (UID: "b7032ceb-75ec-4f69-b174-338e0b964f07"). InnerVolumeSpecName "kube-api-access-fjx7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:13:34.130179 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.130092    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-maas-api-tls" pod "b7032ceb-75ec-4f69-b174-338e0b964f07" (UID: "b7032ceb-75ec-4f69-b174-338e0b964f07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:13:34.130179 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.130152    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-metrics-certs" pod "b7032ceb-75ec-4f69-b174-338e0b964f07" (UID: "b7032ceb-75ec-4f69-b174-338e0b964f07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:13:34.229616 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.229575    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:34.229616 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.229611    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7032ceb-75ec-4f69-b174-338e0b964f07-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:34.229616 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.229623    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7032ceb-75ec-4f69-b174-338e0b964f07-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:34.229876 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.229633    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjx7q\" (UniqueName: \"kubernetes.io/projected/b7032ceb-75ec-4f69-b174-338e0b964f07-kube-api-access-fjx7q\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:34.461116 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.461023    5663 generic.go:356] "Generic (PLEG): container finished" podID="b7032ceb-75ec-4f69-b174-338e0b964f07" containerID="d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a" exitCode=0
Oct 31 13:13:34.461116 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.461072    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"
Oct 31 13:13:34.461116 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.461107    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw" event={"ID":"b7032ceb-75ec-4f69-b174-338e0b964f07","Type":"ContainerDied","Data":"d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a"}
Oct 31 13:13:34.461347 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.461166    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw" event={"ID":"b7032ceb-75ec-4f69-b174-338e0b964f07","Type":"ContainerDied","Data":"8f3ea37dc4568962ddfe48776e9f207a67628527a3123c25e6fdf3213b3289cf"}
Oct 31 13:13:34.461347 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.461193    5663 scope.go:122] "RemoveContainer" containerID="d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a"
Oct 31 13:13:34.470318 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.470294    5663 scope.go:122] "RemoveContainer" containerID="d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a"
Oct 31 13:13:34.470618 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:13:34.470588    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a\": container with ID starting with d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a not found: ID does not exist" containerID="d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a"
Oct 31 13:13:34.470666 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.470614    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a"} err="failed to get container status \"d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a\": rpc error: code = NotFound desc = could not find container \"d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a\": container with ID starting with d55a30befcb93b2a38a55c3c1625bcdbdf4046939a1b04e21570c434c7e8328a not found: ID does not exist"
Oct 31 13:13:34.484177 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.484149    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"]
Oct 31 13:13:34.487964 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:34.487940    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bbf78971-dddf54d77-24lzw"]
Oct 31 13:13:35.436670 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:35.436638    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7032ceb-75ec-4f69-b174-338e0b964f07" path="/var/lib/kubelet/pods/b7032ceb-75ec-4f69-b174-338e0b964f07/volumes"
Oct 31 13:13:38.464468 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.464422    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl" podUID="808b8f1a-2d96-4b02-964e-1558e20bd57d" containerName="payload-processing" containerID="cri-o://249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a" gracePeriod=25
Oct 31 13:13:38.481245 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.481216    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv" podUID="6959288f-5224-4d72-86a9-3063ea6a59af" containerName="payload-pre-processing" containerID="cri-o://40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4" gracePeriod=25
Oct 31 13:13:38.725717 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.725649    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"
Oct 31 13:13:38.754492 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.754460    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"
Oct 31 13:13:38.871660 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.871629    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/808b8f1a-2d96-4b02-964e-1558e20bd57d-kube-api-access-9bhpx\" (UniqueName: \"kubernetes.io/projected/808b8f1a-2d96-4b02-964e-1558e20bd57d-kube-api-access-9bhpx\") pod \"808b8f1a-2d96-4b02-964e-1558e20bd57d\" (UID: \"808b8f1a-2d96-4b02-964e-1558e20bd57d\") "
Oct 31 13:13:38.871840 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.871675    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6959288f-5224-4d72-86a9-3063ea6a59af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6959288f-5224-4d72-86a9-3063ea6a59af-plugins-config-volume\") pod \"6959288f-5224-4d72-86a9-3063ea6a59af\" (UID: \"6959288f-5224-4d72-86a9-3063ea6a59af\") "
Oct 31 13:13:38.871840 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.871784    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6959288f-5224-4d72-86a9-3063ea6a59af-kube-api-access-6x6zw\" (UniqueName: \"kubernetes.io/projected/6959288f-5224-4d72-86a9-3063ea6a59af-kube-api-access-6x6zw\") pod \"6959288f-5224-4d72-86a9-3063ea6a59af\" (UID: \"6959288f-5224-4d72-86a9-3063ea6a59af\") "
Oct 31 13:13:38.871980 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.871841    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/808b8f1a-2d96-4b02-964e-1558e20bd57d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/808b8f1a-2d96-4b02-964e-1558e20bd57d-plugins-config-volume\") pod \"808b8f1a-2d96-4b02-964e-1558e20bd57d\" (UID: \"808b8f1a-2d96-4b02-964e-1558e20bd57d\") "
Oct 31 13:13:38.872211 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.872182    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6959288f-5224-4d72-86a9-3063ea6a59af-plugins-config-volume" pod "6959288f-5224-4d72-86a9-3063ea6a59af" (UID: "6959288f-5224-4d72-86a9-3063ea6a59af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:13:38.872326 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.872217    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/808b8f1a-2d96-4b02-964e-1558e20bd57d-plugins-config-volume" pod "808b8f1a-2d96-4b02-964e-1558e20bd57d" (UID: "808b8f1a-2d96-4b02-964e-1558e20bd57d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:13:38.873836 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.873815    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6959288f-5224-4d72-86a9-3063ea6a59af-kube-api-access-6x6zw" pod "6959288f-5224-4d72-86a9-3063ea6a59af" (UID: "6959288f-5224-4d72-86a9-3063ea6a59af"). InnerVolumeSpecName "kube-api-access-6x6zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:13:38.873900 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.873836    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808b8f1a-2d96-4b02-964e-1558e20bd57d-kube-api-access-9bhpx" pod "808b8f1a-2d96-4b02-964e-1558e20bd57d" (UID: "808b8f1a-2d96-4b02-964e-1558e20bd57d"). InnerVolumeSpecName "kube-api-access-9bhpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:13:38.973623 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.973588    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/808b8f1a-2d96-4b02-964e-1558e20bd57d-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:38.973623 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.973619    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bhpx\" (UniqueName: \"kubernetes.io/projected/808b8f1a-2d96-4b02-964e-1558e20bd57d-kube-api-access-9bhpx\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:38.973623 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.973630    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6959288f-5224-4d72-86a9-3063ea6a59af-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:38.974019 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:38.973641    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x6zw\" (UniqueName: \"kubernetes.io/projected/6959288f-5224-4d72-86a9-3063ea6a59af-kube-api-access-6x6zw\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:39.483899 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.483871    5663 generic.go:356] "Generic (PLEG): container finished" podID="808b8f1a-2d96-4b02-964e-1558e20bd57d" containerID="249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a" exitCode=0
Oct 31 13:13:39.484282 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.483918    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"
Oct 31 13:13:39.484282 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.483969    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl" event={"ID":"808b8f1a-2d96-4b02-964e-1558e20bd57d","Type":"ContainerDied","Data":"249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a"}
Oct 31 13:13:39.484282 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.484040    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl" event={"ID":"808b8f1a-2d96-4b02-964e-1558e20bd57d","Type":"ContainerDied","Data":"00717e5b5ff873579f95acde0265925f0f6b743a222bd9ae3cb5718e440131a1"}
Oct 31 13:13:39.484282 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.484076    5663 scope.go:122] "RemoveContainer" containerID="249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a"
Oct 31 13:13:39.485398 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.485297    5663 generic.go:356] "Generic (PLEG): container finished" podID="6959288f-5224-4d72-86a9-3063ea6a59af" containerID="40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4" exitCode=0
Oct 31 13:13:39.485398 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.485349    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv" event={"ID":"6959288f-5224-4d72-86a9-3063ea6a59af","Type":"ContainerDied","Data":"40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4"}
Oct 31 13:13:39.485398 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.485345    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"
Oct 31 13:13:39.485398 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.485393    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv" event={"ID":"6959288f-5224-4d72-86a9-3063ea6a59af","Type":"ContainerDied","Data":"334dea71f74b670750e33eb5e5899c4324e4bfc4950a16311df7dd3e6593dd68"}
Oct 31 13:13:39.492642 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.492618    5663 scope.go:122] "RemoveContainer" containerID="249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a"
Oct 31 13:13:39.492924 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:13:39.492896    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a\": container with ID starting with 249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a not found: ID does not exist" containerID="249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a"
Oct 31 13:13:39.493022 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.492926    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a"} err="failed to get container status \"249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a\": rpc error: code = NotFound desc = could not find container \"249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a\": container with ID starting with 249f50004f645fbb57c1e0b277c81081e6de998259555bcb6c47c48bfdbf5e3a not found: ID does not exist"
Oct 31 13:13:39.493022 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.492942    5663 scope.go:122] "RemoveContainer" containerID="40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4"
Oct 31 13:13:39.500470 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.500452    5663 scope.go:122] "RemoveContainer" containerID="40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4"
Oct 31 13:13:39.500748 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:13:39.500729    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4\": container with ID starting with 40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4 not found: ID does not exist" containerID="40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4"
Oct 31 13:13:39.500796 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.500756    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4"} err="failed to get container status \"40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4\": rpc error: code = NotFound desc = could not find container \"40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4\": container with ID starting with 40dd62ec0a83584275c96d4927404d7a911e2d02d748e4cb9e1d48cd1d8b56c4 not found: ID does not exist"
Oct 31 13:13:39.505120 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.505095    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"]
Oct 31 13:13:39.510292 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.510272    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bbf78971-96cdc6759-zr7nl"]
Oct 31 13:13:39.520696 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.520676    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"]
Oct 31 13:13:39.523285 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:39.523264    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbf78971-65546bf987-kxwvv"]
Oct 31 13:13:41.213611 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.213571    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"]
Oct 31 13:13:41.214702 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.214662    5663 cpu_manager.go:414] "containerMap: removing container" podUID="b7032ceb-75ec-4f69-b174-338e0b964f07" containerName="maas-api"
Oct 31 13:13:41.214702 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.214691    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7032ceb-75ec-4f69-b174-338e0b964f07" containerName="maas-api"
Oct 31 13:13:41.214932 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.214726    5663 cpu_manager.go:414] "containerMap: removing container" podUID="808b8f1a-2d96-4b02-964e-1558e20bd57d" containerName="payload-processing"
Oct 31 13:13:41.214932 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.214735    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="808b8f1a-2d96-4b02-964e-1558e20bd57d" containerName="payload-processing"
Oct 31 13:13:41.214932 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.214756    5663 cpu_manager.go:414] "containerMap: removing container" podUID="6959288f-5224-4d72-86a9-3063ea6a59af" containerName="payload-pre-processing"
Oct 31 13:13:41.214932 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.214767    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6959288f-5224-4d72-86a9-3063ea6a59af" containerName="payload-pre-processing"
Oct 31 13:13:41.214932 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.214852    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="808b8f1a-2d96-4b02-964e-1558e20bd57d" containerName="payload-processing"
Oct 31 13:13:41.214932 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.214866    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="6959288f-5224-4d72-86a9-3063ea6a59af" containerName="payload-pre-processing"
Oct 31 13:13:41.214932 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.214879    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7032ceb-75ec-4f69-b174-338e0b964f07" containerName="maas-api"
Oct 31 13:13:41.219733 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.219707    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.228542 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.228507    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-198ea4-openshift-default-dockercfg-jwdt9\""
Oct 31 13:13:41.230655 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.230626    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"]
Oct 31 13:13:41.294629 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.294586    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-envoy\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.294629 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.294628    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-data\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.294869 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.294652    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-podinfo\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.294869 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.294732    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istiod-ca-cert\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.294869 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.294829    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbpw\" (UniqueName: \"kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-kube-api-access-wnbpw\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.294997 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.294877    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-token\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.294997 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.294901    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-certs\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.294997 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.294973    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-socket\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.295114 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.295004    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-credential-socket\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.395767 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.395730    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbpw\" (UniqueName: \"kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-kube-api-access-wnbpw\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.395767 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.395773    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-token\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.396067 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.395794    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-certs\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.396067 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.395837    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-socket\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.396067 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.395877    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-credential-socket\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.396067 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.395964    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-envoy\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.396067 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.395988    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-data\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.396067 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.396011    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-podinfo\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.396067 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.396039    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istiod-ca-cert\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.396479 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.396387    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-certs\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.396546 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.396484    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-socket\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.396601 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.396545    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-data\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.396721 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.396699    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-credential-socket\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.396886 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.396870    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istiod-ca-cert\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.398873 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.398851    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-envoy\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.399097 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.399061    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-podinfo\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.408029 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.407997    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-token\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.408469 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.408447    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbpw\" (UniqueName: \"kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-kube-api-access-wnbpw\") pod \"e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") " pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.437425 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.437391    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6959288f-5224-4d72-86a9-3063ea6a59af" path="/var/lib/kubelet/pods/6959288f-5224-4d72-86a9-3063ea6a59af/volumes"
Oct 31 13:13:41.437858 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.437838    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="808b8f1a-2d96-4b02-964e-1558e20bd57d" path="/var/lib/kubelet/pods/808b8f1a-2d96-4b02-964e-1558e20bd57d/volumes"
Oct 31 13:13:41.535501 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.535411    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:41.816709 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.816678    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"]
Oct 31 13:13:41.819835 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:13:41.819807    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ddde61_7a63_46a2_80a6_aaa3588e0c08.slice/crio-9e55b586c53a44f0347a0dc67edd97f9e0c83450b81bf23d6d555e4cb7931f68 WatchSource:0}: Error finding container 9e55b586c53a44f0347a0dc67edd97f9e0c83450b81bf23d6d555e4cb7931f68: Status 404 returned error can't find the container with id 9e55b586c53a44f0347a0dc67edd97f9e0c83450b81bf23d6d555e4cb7931f68
Oct 31 13:13:41.821831 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.821798    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:13:41.821923 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.821861    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:13:41.821923 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:41.821892    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:13:42.502284 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:42.502251    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" event={"ID":"98ddde61-7a63-46a2-80a6-aaa3588e0c08","Type":"ContainerStarted","Data":"c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f"}
Oct 31 13:13:42.502696 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:42.502298    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" event={"ID":"98ddde61-7a63-46a2-80a6-aaa3588e0c08","Type":"ContainerStarted","Data":"9e55b586c53a44f0347a0dc67edd97f9e0c83450b81bf23d6d555e4cb7931f68"}
Oct 31 13:13:42.526923 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:42.526840    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" podStartSLOduration=1.5268209929999998 podStartE2EDuration="1.526820993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:42.526043707 +0000 UTC m=+2747.266469925" watchObservedRunningTime="2026-10-31 13:13:42.526820993 +0000 UTC m=+2747.267247209"
Oct 31 13:13:42.536279 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:42.536254    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:42.541809 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:42.541788    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:43.506441 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:43.506410    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:43.507412 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:43.507390    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:13:45.972137 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:45.972096    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"]
Oct 31 13:13:45.975743 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:45.975724    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:45.991531 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:45.991506    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-198ea4-serving-cert\""
Oct 31 13:13:46.001070 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.001040    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"]
Oct 31 13:13:46.090242 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.090203    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"]
Oct 31 13:13:46.093597 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.093580    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"
Oct 31 13:13:46.111562 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.111530    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"]
Oct 31 13:13:46.112049 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.112030    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-198ea4\""
Oct 31 13:13:46.140977 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.140940    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-metrics-certs\") pod \"maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.141144 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.140988    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh6kd\" (UniqueName: \"kubernetes.io/projected/ea478ebe-2d3d-4d38-a65d-d913915e532a-kube-api-access-xh6kd\") pod \"maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.141194 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.141134    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.141275 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.141258    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea478ebe-2d3d-4d38-a65d-d913915e532a-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.231077 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.230979    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"]
Oct 31 13:13:46.235398 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.235371    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"
Oct 31 13:13:46.242466 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.242318    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea478ebe-2d3d-4d38-a65d-d913915e532a-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.242857 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.242486    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-metrics-certs\") pod \"maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.242857 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.242534    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwbsg\" (UniqueName: \"kubernetes.io/projected/572e6bfe-c3af-4842-94fd-d5292b5d7092-kube-api-access-mwbsg\") pod \"payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8\" (UID: \"572e6bfe-c3af-4842-94fd-d5292b5d7092\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"
Oct 31 13:13:46.242857 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.242562    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh6kd\" (UniqueName: \"kubernetes.io/projected/ea478ebe-2d3d-4d38-a65d-d913915e532a-kube-api-access-xh6kd\") pod \"maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.242857 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.242619    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.242857 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.242653    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572e6bfe-c3af-4842-94fd-d5292b5d7092-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8\" (UID: \"572e6bfe-c3af-4842-94fd-d5292b5d7092\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"
Oct 31 13:13:46.243505 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.243478    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea478ebe-2d3d-4d38-a65d-d913915e532a-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.248071 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.246953    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.248071 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.247581    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-198ea4-dockercfg-mhldq\""
Oct 31 13:13:46.258971 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.258923    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"]
Oct 31 13:13:46.259709 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.259687    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh6kd\" (UniqueName: \"kubernetes.io/projected/ea478ebe-2d3d-4d38-a65d-d913915e532a-kube-api-access-xh6kd\") pod \"maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.259880 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.259857    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-metrics-certs\") pod \"maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.289658 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.289629    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:46.344684 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.344641    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572e6bfe-c3af-4842-94fd-d5292b5d7092-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8\" (UID: \"572e6bfe-c3af-4842-94fd-d5292b5d7092\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"
Oct 31 13:13:46.344793 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.344743    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32cdc561-19d5-4737-9a65-afc1fe919638-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt\" (UID: \"32cdc561-19d5-4737-9a65-afc1fe919638\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"
Oct 31 13:13:46.344856 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.344796    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvp4\" (UniqueName: \"kubernetes.io/projected/32cdc561-19d5-4737-9a65-afc1fe919638-kube-api-access-wsvp4\") pod \"payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt\" (UID: \"32cdc561-19d5-4737-9a65-afc1fe919638\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"
Oct 31 13:13:46.344906 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.344894    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwbsg\" (UniqueName: \"kubernetes.io/projected/572e6bfe-c3af-4842-94fd-d5292b5d7092-kube-api-access-mwbsg\") pod \"payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8\" (UID: \"572e6bfe-c3af-4842-94fd-d5292b5d7092\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"
Oct 31 13:13:46.345547 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.345522    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572e6bfe-c3af-4842-94fd-d5292b5d7092-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8\" (UID: \"572e6bfe-c3af-4842-94fd-d5292b5d7092\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"
Oct 31 13:13:46.361907 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.361879    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwbsg\" (UniqueName: \"kubernetes.io/projected/572e6bfe-c3af-4842-94fd-d5292b5d7092-kube-api-access-mwbsg\") pod \"payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8\" (UID: \"572e6bfe-c3af-4842-94fd-d5292b5d7092\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"
Oct 31 13:13:46.406522 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.406495    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"
Oct 31 13:13:46.445838 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.445800    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvp4\" (UniqueName: \"kubernetes.io/projected/32cdc561-19d5-4737-9a65-afc1fe919638-kube-api-access-wsvp4\") pod \"payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt\" (UID: \"32cdc561-19d5-4737-9a65-afc1fe919638\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"
Oct 31 13:13:46.446032 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.445942    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32cdc561-19d5-4737-9a65-afc1fe919638-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt\" (UID: \"32cdc561-19d5-4737-9a65-afc1fe919638\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"
Oct 31 13:13:46.446725 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.446702    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32cdc561-19d5-4737-9a65-afc1fe919638-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt\" (UID: \"32cdc561-19d5-4737-9a65-afc1fe919638\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"
Oct 31 13:13:46.463527 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.463283    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvp4\" (UniqueName: \"kubernetes.io/projected/32cdc561-19d5-4737-9a65-afc1fe919638-kube-api-access-wsvp4\") pod \"payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt\" (UID: \"32cdc561-19d5-4737-9a65-afc1fe919638\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"
Oct 31 13:13:46.565786 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.565735    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"]
Oct 31 13:13:46.581555 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.581525    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"
Oct 31 13:13:46.689640 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.689613    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"]
Oct 31 13:13:46.692161 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:13:46.692132    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572e6bfe_c3af_4842_94fd_d5292b5d7092.slice/crio-68613b9ddff19d20bf1c5f59e27ae8b85b758a6d32c3c6d78e9e016d56bdc109 WatchSource:0}: Error finding container 68613b9ddff19d20bf1c5f59e27ae8b85b758a6d32c3c6d78e9e016d56bdc109: Status 404 returned error can't find the container with id 68613b9ddff19d20bf1c5f59e27ae8b85b758a6d32c3c6d78e9e016d56bdc109
Oct 31 13:13:46.860813 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:46.860736    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"]
Oct 31 13:13:46.867995 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:13:46.867973    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cdc561_19d5_4737_9a65_afc1fe919638.slice/crio-33404a8993792e3e53b91cdcd9add310c412ce0d5f343596dddad2fc5f63d8da WatchSource:0}: Error finding container 33404a8993792e3e53b91cdcd9add310c412ce0d5f343596dddad2fc5f63d8da: Status 404 returned error can't find the container with id 33404a8993792e3e53b91cdcd9add310c412ce0d5f343596dddad2fc5f63d8da
Oct 31 13:13:47.526486 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:47.526444    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w" event={"ID":"ea478ebe-2d3d-4d38-a65d-d913915e532a","Type":"ContainerStarted","Data":"34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999"}
Oct 31 13:13:47.526919 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:47.526510    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:47.526919 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:47.526530    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w" event={"ID":"ea478ebe-2d3d-4d38-a65d-d913915e532a","Type":"ContainerStarted","Data":"ade42b0eefe5c87b3215fe46715516ffcd23df9d683b6e4bc07f9065bf597702"}
Oct 31 13:13:47.528018 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:47.527996    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8" event={"ID":"572e6bfe-c3af-4842-94fd-d5292b5d7092","Type":"ContainerStarted","Data":"d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec"}
Oct 31 13:13:47.528116 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:47.528039    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8" event={"ID":"572e6bfe-c3af-4842-94fd-d5292b5d7092","Type":"ContainerStarted","Data":"68613b9ddff19d20bf1c5f59e27ae8b85b758a6d32c3c6d78e9e016d56bdc109"}
Oct 31 13:13:47.528116 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:47.528068    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"
Oct 31 13:13:47.529485 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:47.529460    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt" event={"ID":"32cdc561-19d5-4737-9a65-afc1fe919638","Type":"ContainerStarted","Data":"e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20"}
Oct 31 13:13:47.529617 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:47.529513    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt" event={"ID":"32cdc561-19d5-4737-9a65-afc1fe919638","Type":"ContainerStarted","Data":"33404a8993792e3e53b91cdcd9add310c412ce0d5f343596dddad2fc5f63d8da"}
Oct 31 13:13:47.529617 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:47.529577    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"
Oct 31 13:13:47.551773 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:47.551732    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w" podStartSLOduration=2.14463791 podStartE2EDuration="2.551719423s" totalImagesPullingTime="407.081513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:47.550745968 +0000 UTC m=+2752.291172186" watchObservedRunningTime="2026-10-31 13:13:47.551719423 +0000 UTC m=+2752.292145638"
Oct 31 13:13:47.570982 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:47.570942    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt" podStartSLOduration=1.570930487 podStartE2EDuration="1.570930487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:47.569912695 +0000 UTC m=+2752.310338911" watchObservedRunningTime="2026-10-31 13:13:47.570930487 +0000 UTC m=+2752.311356702"
Oct 31 13:13:47.588334 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:47.588283    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8" podStartSLOduration=1.5882692349999998 podStartE2EDuration="1.588269235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:47.586997519 +0000 UTC m=+2752.327423736" watchObservedRunningTime="2026-10-31 13:13:47.588269235 +0000 UTC m=+2752.328695453"
Oct 31 13:13:52.493770 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:13:52.493733    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:13:53.540373 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:53.540318    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:13:58.508690 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:58.508653    5663 patch_prober.go:29] interesting pod/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 31 13:13:58.509130 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:58.508703    5663 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 31 13:13:58.537250 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:58.537220    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"
Oct 31 13:13:58.537646 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:58.537628    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"
Oct 31 13:13:58.588592 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:58.588560    5663 generic.go:356] "Generic (PLEG): container finished" podID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerID="c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f" exitCode=0
Oct 31 13:13:58.588769 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:58.588607    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" event={"ID":"98ddde61-7a63-46a2-80a6-aaa3588e0c08","Type":"ContainerDied","Data":"c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f"}
Oct 31 13:13:58.588977 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:58.588961    5663 scope.go:122] "RemoveContainer" containerID="c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f"
Oct 31 13:13:58.590230 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:58.590191    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:13:58.590318 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:58.590291    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:13:58.590393 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:58.590373    5663 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 13:13:59.594910 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:59.594878    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" event={"ID":"98ddde61-7a63-46a2-80a6-aaa3588e0c08","Type":"ContainerStarted","Data":"4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b"}
Oct 31 13:13:59.842425 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:13:59.842388    5663 scope.go:122] "RemoveContainer" containerID="3c9ddbb63a76b463d85da568903dc4b45f9592c21556be07f076337104613f6e"
Oct 31 13:14:00.536034 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:00.535990    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:14:00.536255 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:00.536233    5663 patch_prober.go:29] interesting pod/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 31 13:14:00.536339 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:00.536291    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 31 13:14:01.536545 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:01.536507    5663 patch_prober.go:29] interesting pod/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 31 13:14:01.536971 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:01.536563    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 31 13:14:02.536222 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:02.536185    5663 patch_prober.go:29] interesting pod/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 31 13:14:02.536474 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:02.536239    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 31 13:14:03.535887 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:03.535848    5663 patch_prober.go:29] interesting pod/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 31 13:14:03.536434 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:03.535899    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 31 13:14:04.536098 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:04.536066    5663 patch_prober.go:29] interesting pod/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 31 13:14:04.536607 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:04.536131    5663 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 31 13:14:05.540637 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:05.540608    5663 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:14:05.541055 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:05.541016    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:14:05.541801 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:05.541779    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:14:22.475303 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:14:22.475267    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:14:44.022393 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:44.022288    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"]
Oct 31 13:14:44.028439 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:44.028406    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rp6g4"]
Oct 31 13:14:45.438026 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:14:45.437992    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fcbaba0-4c84-4304-8b14-92d668b84b3e" path="/var/lib/kubelet/pods/9fcbaba0-4c84-4304-8b14-92d668b84b3e/volumes"
Oct 31 13:14:52.474955 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:14:52.474912    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:15:03.034189 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:03.034140    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"]
Oct 31 13:15:03.039754 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:03.039720    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-cz9v2"]
Oct 31 13:15:03.437830 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:03.437794    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b2ac3df-7d7f-403d-a723-83a1eca1969d" path="/var/lib/kubelet/pods/8b2ac3df-7d7f-403d-a723-83a1eca1969d/volumes"
Oct 31 13:15:22.476063 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:15:22.476009    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:15:32.616392 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:32.615296    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"]
Oct 31 13:15:32.620196 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:32.620162    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"]
Oct 31 13:15:37.615790 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:37.615744    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8" podUID="572e6bfe-c3af-4842-94fd-d5292b5d7092" containerName="payload-pre-processing" containerID="cri-o://d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec" gracePeriod=25
Oct 31 13:15:37.621281 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:37.621252    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt" podUID="32cdc561-19d5-4737-9a65-afc1fe919638" containerName="payload-processing" containerID="cri-o://e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20" gracePeriod=25
Oct 31 13:15:37.869662 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:37.869590    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"
Oct 31 13:15:37.909445 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:37.909336    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"
Oct 31 13:15:37.915286 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:37.915261    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572e6bfe-c3af-4842-94fd-d5292b5d7092-kube-api-access-mwbsg\" (UniqueName: \"kubernetes.io/projected/572e6bfe-c3af-4842-94fd-d5292b5d7092-kube-api-access-mwbsg\") pod \"572e6bfe-c3af-4842-94fd-d5292b5d7092\" (UID: \"572e6bfe-c3af-4842-94fd-d5292b5d7092\") "
Oct 31 13:15:37.915431 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:37.915296    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/572e6bfe-c3af-4842-94fd-d5292b5d7092-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572e6bfe-c3af-4842-94fd-d5292b5d7092-plugins-config-volume\") pod \"572e6bfe-c3af-4842-94fd-d5292b5d7092\" (UID: \"572e6bfe-c3af-4842-94fd-d5292b5d7092\") "
Oct 31 13:15:37.915687 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:37.915666    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572e6bfe-c3af-4842-94fd-d5292b5d7092-plugins-config-volume" pod "572e6bfe-c3af-4842-94fd-d5292b5d7092" (UID: "572e6bfe-c3af-4842-94fd-d5292b5d7092"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:37.917394 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:37.917375    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572e6bfe-c3af-4842-94fd-d5292b5d7092-kube-api-access-mwbsg" pod "572e6bfe-c3af-4842-94fd-d5292b5d7092" (UID: "572e6bfe-c3af-4842-94fd-d5292b5d7092"). InnerVolumeSpecName "kube-api-access-mwbsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:38.016433 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.016389    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32cdc561-19d5-4737-9a65-afc1fe919638-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32cdc561-19d5-4737-9a65-afc1fe919638-plugins-config-volume\") pod \"32cdc561-19d5-4737-9a65-afc1fe919638\" (UID: \"32cdc561-19d5-4737-9a65-afc1fe919638\") "
Oct 31 13:15:38.016630 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.016524    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32cdc561-19d5-4737-9a65-afc1fe919638-kube-api-access-wsvp4\" (UniqueName: \"kubernetes.io/projected/32cdc561-19d5-4737-9a65-afc1fe919638-kube-api-access-wsvp4\") pod \"32cdc561-19d5-4737-9a65-afc1fe919638\" (UID: \"32cdc561-19d5-4737-9a65-afc1fe919638\") "
Oct 31 13:15:38.016798 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.016775    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cdc561-19d5-4737-9a65-afc1fe919638-plugins-config-volume" pod "32cdc561-19d5-4737-9a65-afc1fe919638" (UID: "32cdc561-19d5-4737-9a65-afc1fe919638"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:38.016868 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.016853    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32cdc561-19d5-4737-9a65-afc1fe919638-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:38.016907 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.016874    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwbsg\" (UniqueName: \"kubernetes.io/projected/572e6bfe-c3af-4842-94fd-d5292b5d7092-kube-api-access-mwbsg\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:38.016907 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.016887    5663 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572e6bfe-c3af-4842-94fd-d5292b5d7092-plugins-config-volume\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:38.018560 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.018535    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cdc561-19d5-4737-9a65-afc1fe919638-kube-api-access-wsvp4" pod "32cdc561-19d5-4737-9a65-afc1fe919638" (UID: "32cdc561-19d5-4737-9a65-afc1fe919638"). InnerVolumeSpecName "kube-api-access-wsvp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:38.042578 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.042542    5663 generic.go:356] "Generic (PLEG): container finished" podID="572e6bfe-c3af-4842-94fd-d5292b5d7092" containerID="d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec" exitCode=0
Oct 31 13:15:38.042747 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.042588    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"
Oct 31 13:15:38.042747 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.042631    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8" event={"ID":"572e6bfe-c3af-4842-94fd-d5292b5d7092","Type":"ContainerDied","Data":"d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec"}
Oct 31 13:15:38.042747 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.042683    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8" event={"ID":"572e6bfe-c3af-4842-94fd-d5292b5d7092","Type":"ContainerDied","Data":"68613b9ddff19d20bf1c5f59e27ae8b85b758a6d32c3c6d78e9e016d56bdc109"}
Oct 31 13:15:38.042747 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.042725    5663 scope.go:122] "RemoveContainer" containerID="d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec"
Oct 31 13:15:38.044096 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.044072    5663 generic.go:356] "Generic (PLEG): container finished" podID="32cdc561-19d5-4737-9a65-afc1fe919638" containerID="e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20" exitCode=0
Oct 31 13:15:38.044231 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.044135    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"
Oct 31 13:15:38.044231 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.044156    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt" event={"ID":"32cdc561-19d5-4737-9a65-afc1fe919638","Type":"ContainerDied","Data":"e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20"}
Oct 31 13:15:38.044231 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.044201    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt" event={"ID":"32cdc561-19d5-4737-9a65-afc1fe919638","Type":"ContainerDied","Data":"33404a8993792e3e53b91cdcd9add310c412ce0d5f343596dddad2fc5f63d8da"}
Oct 31 13:15:38.062019 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.061836    5663 scope.go:122] "RemoveContainer" containerID="d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec"
Oct 31 13:15:38.062419 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:15:38.062390    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec\": container with ID starting with d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec not found: ID does not exist" containerID="d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec"
Oct 31 13:15:38.062554 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.062424    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec"} err="failed to get container status \"d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec\": rpc error: code = NotFound desc = could not find container \"d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec\": container with ID starting with d4338d7535e4554088510b38fd0c7bc28bd242ab80ac50702adcbfcd4d46c9ec not found: ID does not exist"
Oct 31 13:15:38.062554 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.062440    5663 scope.go:122] "RemoveContainer" containerID="e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20"
Oct 31 13:15:38.070561 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.070437    5663 scope.go:122] "RemoveContainer" containerID="e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20"
Oct 31 13:15:38.070791 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:15:38.070766    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20\": container with ID starting with e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20 not found: ID does not exist" containerID="e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20"
Oct 31 13:15:38.070883 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.070799    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20"} err="failed to get container status \"e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20\": rpc error: code = NotFound desc = could not find container \"e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20\": container with ID starting with e0d1f6c758f59362df452630a9536bd1bcd9f69a5b4556756be87abcee9afd20 not found: ID does not exist"
Oct 31 13:15:38.071964 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.071940    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"]
Oct 31 13:15:38.079575 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.079553    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-198ea4-c84999559-6b7jt"]
Oct 31 13:15:38.091646 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.091621    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"]
Oct 31 13:15:38.098690 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.098667    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-198ea4-8777cb4ff-dwzr8"]
Oct 31 13:15:38.107892 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.107865    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"]
Oct 31 13:15:38.108113 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.108090    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w" podUID="ea478ebe-2d3d-4d38-a65d-d913915e532a" containerName="maas-api" containerID="cri-o://34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999" gracePeriod=30
Oct 31 13:15:38.117782 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.117751    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsvp4\" (UniqueName: \"kubernetes.io/projected/32cdc561-19d5-4737-9a65-afc1fe919638-kube-api-access-wsvp4\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:38.345626 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.345600    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:15:38.420972 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.420882    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea478ebe-2d3d-4d38-a65d-d913915e532a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea478ebe-2d3d-4d38-a65d-d913915e532a-openshift-service-ca\") pod \"ea478ebe-2d3d-4d38-a65d-d913915e532a\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") "
Oct 31 13:15:38.420972 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.420967    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-maas-api-tls\") pod \"ea478ebe-2d3d-4d38-a65d-d913915e532a\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") "
Oct 31 13:15:38.421253 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.420996    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-metrics-certs\") pod \"ea478ebe-2d3d-4d38-a65d-d913915e532a\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") "
Oct 31 13:15:38.421253 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.421085    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea478ebe-2d3d-4d38-a65d-d913915e532a-kube-api-access-xh6kd\" (UniqueName: \"kubernetes.io/projected/ea478ebe-2d3d-4d38-a65d-d913915e532a-kube-api-access-xh6kd\") pod \"ea478ebe-2d3d-4d38-a65d-d913915e532a\" (UID: \"ea478ebe-2d3d-4d38-a65d-d913915e532a\") "
Oct 31 13:15:38.421379 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.421330    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea478ebe-2d3d-4d38-a65d-d913915e532a-openshift-service-ca" pod "ea478ebe-2d3d-4d38-a65d-d913915e532a" (UID: "ea478ebe-2d3d-4d38-a65d-d913915e532a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:38.423300 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.423267    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-metrics-certs" pod "ea478ebe-2d3d-4d38-a65d-d913915e532a" (UID: "ea478ebe-2d3d-4d38-a65d-d913915e532a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:38.423300 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.423284    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea478ebe-2d3d-4d38-a65d-d913915e532a-kube-api-access-xh6kd" pod "ea478ebe-2d3d-4d38-a65d-d913915e532a" (UID: "ea478ebe-2d3d-4d38-a65d-d913915e532a"). InnerVolumeSpecName "kube-api-access-xh6kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:38.423300 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.423285    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-maas-api-tls" pod "ea478ebe-2d3d-4d38-a65d-d913915e532a" (UID: "ea478ebe-2d3d-4d38-a65d-d913915e532a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:38.522599 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.522545    5663 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-maas-api-tls\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:38.522599 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.522593    5663 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea478ebe-2d3d-4d38-a65d-d913915e532a-metrics-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:38.522599 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.522605    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh6kd\" (UniqueName: \"kubernetes.io/projected/ea478ebe-2d3d-4d38-a65d-d913915e532a-kube-api-access-xh6kd\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:38.522599 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:38.522615    5663 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea478ebe-2d3d-4d38-a65d-d913915e532a-openshift-service-ca\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:39.052221 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:39.052185    5663 generic.go:356] "Generic (PLEG): container finished" podID="ea478ebe-2d3d-4d38-a65d-d913915e532a" containerID="34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999" exitCode=0
Oct 31 13:15:39.052706 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:39.052229    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w" event={"ID":"ea478ebe-2d3d-4d38-a65d-d913915e532a","Type":"ContainerDied","Data":"34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999"}
Oct 31 13:15:39.052706 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:39.052272    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w" event={"ID":"ea478ebe-2d3d-4d38-a65d-d913915e532a","Type":"ContainerDied","Data":"ade42b0eefe5c87b3215fe46715516ffcd23df9d683b6e4bc07f9065bf597702"}
Oct 31 13:15:39.052706 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:39.052294    5663 scope.go:122] "RemoveContainer" containerID="34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999"
Oct 31 13:15:39.052706 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:39.052232    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"
Oct 31 13:15:39.061274 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:39.061255    5663 scope.go:122] "RemoveContainer" containerID="34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999"
Oct 31 13:15:39.061675 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:15:39.061653    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999\": container with ID starting with 34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999 not found: ID does not exist" containerID="34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999"
Oct 31 13:15:39.061737 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:39.061685    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999"} err="failed to get container status \"34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999\": rpc error: code = NotFound desc = could not find container \"34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999\": container with ID starting with 34a535c5c0a5d03aa9a6ee4dbf9550fc05bab81ef3d4d56ad5070039e1714999 not found: ID does not exist"
Oct 31 13:15:39.073424 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:39.073395    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"]
Oct 31 13:15:39.075409 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:39.075384    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-198ea4-d88f9b7f6-x4r4w"]
Oct 31 13:15:39.437211 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:39.437180    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32cdc561-19d5-4737-9a65-afc1fe919638" path="/var/lib/kubelet/pods/32cdc561-19d5-4737-9a65-afc1fe919638/volumes"
Oct 31 13:15:39.437588 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:39.437573    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572e6bfe-c3af-4842-94fd-d5292b5d7092" path="/var/lib/kubelet/pods/572e6bfe-c3af-4842-94fd-d5292b5d7092/volumes"
Oct 31 13:15:39.437940 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:39.437927    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea478ebe-2d3d-4d38-a65d-d913915e532a" path="/var/lib/kubelet/pods/ea478ebe-2d3d-4d38-a65d-d913915e532a/volumes"
Oct 31 13:15:45.179664 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:45.179623    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"]
Oct 31 13:15:45.180166 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:45.179943    5663 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy" containerID="cri-o://4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b" gracePeriod=30
Oct 31 13:15:50.426400 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.426375    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:15:50.553175 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553094    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-certs\") pod \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") "
Oct 31 13:15:50.553175 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553156    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istiod-ca-cert\") pod \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") "
Oct 31 13:15:50.553175 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553175    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-token\" (UniqueName: \"kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-token\") pod \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") "
Oct 31 13:15:50.553455 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553260    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-podinfo\") pod \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") "
Oct 31 13:15:50.553455 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553293    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-credential-socket\") pod \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") "
Oct 31 13:15:50.553455 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553334    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-kube-api-access-wnbpw\" (UniqueName: \"kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-kube-api-access-wnbpw\") pod \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") "
Oct 31 13:15:50.553455 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553377    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-data\") pod \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") "
Oct 31 13:15:50.553455 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553431    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-envoy\") pod \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") "
Oct 31 13:15:50.553696 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553446    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-certs" pod "98ddde61-7a63-46a2-80a6-aaa3588e0c08" (UID: "98ddde61-7a63-46a2-80a6-aaa3588e0c08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:50.553696 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553465    5663 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-socket\") pod \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\" (UID: \"98ddde61-7a63-46a2-80a6-aaa3588e0c08\") "
Oct 31 13:15:50.553696 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553688    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-socket" pod "98ddde61-7a63-46a2-80a6-aaa3588e0c08" (UID: "98ddde61-7a63-46a2-80a6-aaa3588e0c08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:50.553862 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553711    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istiod-ca-cert" pod "98ddde61-7a63-46a2-80a6-aaa3588e0c08" (UID: "98ddde61-7a63-46a2-80a6-aaa3588e0c08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:50.553927 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553853    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-data" pod "98ddde61-7a63-46a2-80a6-aaa3588e0c08" (UID: "98ddde61-7a63-46a2-80a6-aaa3588e0c08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:50.553927 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553879    5663 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:50.554040 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553926    5663 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-workload-certs\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:50.554040 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553930    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-credential-socket" pod "98ddde61-7a63-46a2-80a6-aaa3588e0c08" (UID: "98ddde61-7a63-46a2-80a6-aaa3588e0c08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:50.554040 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.553944    5663 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istiod-ca-cert\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:50.555595 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.555568    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-token" pod "98ddde61-7a63-46a2-80a6-aaa3588e0c08" (UID: "98ddde61-7a63-46a2-80a6-aaa3588e0c08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:50.555728 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.555626    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-podinfo" pod "98ddde61-7a63-46a2-80a6-aaa3588e0c08" (UID: "98ddde61-7a63-46a2-80a6-aaa3588e0c08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:15:50.555728 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.555689    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-kube-api-access-wnbpw" pod "98ddde61-7a63-46a2-80a6-aaa3588e0c08" (UID: "98ddde61-7a63-46a2-80a6-aaa3588e0c08"). InnerVolumeSpecName "kube-api-access-wnbpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:50.555812 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.555748    5663 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-envoy" pod "98ddde61-7a63-46a2-80a6-aaa3588e0c08" (UID: "98ddde61-7a63-46a2-80a6-aaa3588e0c08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:50.655315 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.655272    5663 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-token\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:50.655315 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.655307    5663 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-podinfo\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:50.655315 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.655319    5663 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-credential-socket\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:50.655591 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.655329    5663 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnbpw\" (UniqueName: \"kubernetes.io/projected/98ddde61-7a63-46a2-80a6-aaa3588e0c08-kube-api-access-wnbpw\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:50.655591 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.655370    5663 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-data\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:50.655591 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:50.655380    5663 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ddde61-7a63-46a2-80a6-aaa3588e0c08-istio-envoy\") on node \"ip-10-0-140-137.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:51.135979 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.135942    5663 generic.go:356] "Generic (PLEG): container finished" podID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerID="4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b" exitCode=0
Oct 31 13:15:51.136194 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.136003    5663 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"
Oct 31 13:15:51.136194 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.136028    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" event={"ID":"98ddde61-7a63-46a2-80a6-aaa3588e0c08","Type":"ContainerDied","Data":"4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b"}
Oct 31 13:15:51.136194 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.136086    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq" event={"ID":"98ddde61-7a63-46a2-80a6-aaa3588e0c08","Type":"ContainerDied","Data":"9e55b586c53a44f0347a0dc67edd97f9e0c83450b81bf23d6d555e4cb7931f68"}
Oct 31 13:15:51.136194 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.136109    5663 scope.go:122] "RemoveContainer" containerID="4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b"
Oct 31 13:15:51.145141 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.145120    5663 scope.go:122] "RemoveContainer" containerID="c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f"
Oct 31 13:15:51.152914 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.152897    5663 scope.go:122] "RemoveContainer" containerID="4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b"
Oct 31 13:15:51.153183 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:15:51.153165    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b\": container with ID starting with 4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b not found: ID does not exist" containerID="4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b"
Oct 31 13:15:51.153233 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.153192    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b"} err="failed to get container status \"4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b\": rpc error: code = NotFound desc = could not find container \"4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b\": container with ID starting with 4b73ffb1fc566ea32e7d22d8d06ee13dd9b8253a9d999056811e3a86ac57340b not found: ID does not exist"
Oct 31 13:15:51.153233 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.153210    5663 scope.go:122] "RemoveContainer" containerID="c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f"
Oct 31 13:15:51.153459 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:15:51.153439    5663 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f\": container with ID starting with c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f not found: ID does not exist" containerID="c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f"
Oct 31 13:15:51.153528 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.153468    5663 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f"} err="failed to get container status \"c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f\": rpc error: code = NotFound desc = could not find container \"c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f\": container with ID starting with c8d3dcafaf7e05fd29826f9b6b722b165a98ce93d910c575a621f115bb7eb85f not found: ID does not exist"
Oct 31 13:15:51.158029 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.158010    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"]
Oct 31 13:15:51.162878 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.162856    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-198ea4-openshift-default-599fcf889c-hr9vq"]
Oct 31 13:15:51.437074 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:51.436995    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" path="/var/lib/kubelet/pods/98ddde61-7a63-46a2-80a6-aaa3588e0c08/volumes"
Oct 31 13:15:52.475805 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:15:52.475773    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:15:59.909019 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:59.908987    5663 scope.go:122] "RemoveContainer" containerID="4ca37c9735ae8540201c71bd759405981fb7d9e53c1445584f08ad7908718aef"
Oct 31 13:15:59.917180 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:15:59.917160    5663 scope.go:122] "RemoveContainer" containerID="b15807c018aac104629af218566c5a8f59ad0058a69ad754c33a0ad4d02537a2"
Oct 31 13:16:22.474015 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:16:22.473979    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:16:48.459428 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:48.459396    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dwk2j_dd96601d-214f-4e8f-9003-a78e49ba51a2/manager/0.log"
Oct 31 13:16:48.824752 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:48.824719    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-mt6xx_19431d95-2890-4c4c-b7b4-d7adb4fd51d5/manager/0.log"
Oct 31 13:16:49.437282 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:49.437253    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wd2j4_fbac8c23-7b52-45f5-b50f-649c72ac1fd7/manager/0.log"
Oct 31 13:16:49.648201 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:49.648172    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wfkln_fff0b729-f78b-4bd7-ad0a-e66d5b14243d/proxy/0.log"
Oct 31 13:16:50.350036 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:50.350008    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c54444cdb-4pldr_4814e92c-745a-480b-8b21-540d70d85cf6/maas-api/0.log"
Oct 31 13:16:50.895535 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:50.895503    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-8zvrv_8a257c14-3432-4628-b2e2-e2f3cf9f18c1/revoke-keys/0.log"
Oct 31 13:16:51.056968 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:51.056941    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-25pvx_c786f506-d163-4051-8d83-27f982c1c9ea/revoke-keys/0.log"
Oct 31 13:16:51.403598 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:51.403559    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e4f7bebc-7td2r_48a3d515-18f3-4084-8b25-60ca30514d21/revoke-keys/0.log"
Oct 31 13:16:51.755931 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:51.755819    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-wwt7t_8d3eae84-8c5d-4fd2-b983-1d3e34d1507b/postgres/0.log"
Oct 31 13:16:52.475483 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:16:52.475449    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:16:55.633010 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:55.632965    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cf7fb5f5-z927m_631d337f-1016-400a-99df-436efb0db2e0/kube-auth-proxy/0.log"
Oct 31 13:16:56.004580 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:56.004461    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-7q2ln_c8b74799-9fa1-41b6-99b1-6dc9a656841d/istio-proxy/0.log"
Oct 31 13:16:56.524598 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:56.524571    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-dswld_57ad591e-3b45-4312-8612-cf848dcf55e5/istio-proxy/0.log"
Oct 31 13:16:56.868251 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:16:56.868220    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-lsjpx_6cacf32b-0e2d-48cc-a0be-c99e26159bcc/istio-proxy/0.log"
Oct 31 13:17:10.073038 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:10.073010    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dwk2j_dd96601d-214f-4e8f-9003-a78e49ba51a2/manager/0.log"
Oct 31 13:17:10.462281 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:10.462204    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-mt6xx_19431d95-2890-4c4c-b7b4-d7adb4fd51d5/manager/0.log"
Oct 31 13:17:11.023369 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:11.023315    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"]
Oct 31 13:17:11.026504 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:11.026480    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-25pvx"]
Oct 31 13:17:11.047186 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:11.047161    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wd2j4_fbac8c23-7b52-45f5-b50f-649c72ac1fd7/manager/0.log"
Oct 31 13:17:11.249868 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:11.249841    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wfkln_fff0b729-f78b-4bd7-ad0a-e66d5b14243d/proxy/0.log"
Oct 31 13:17:11.437557 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:11.437523    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c786f506-d163-4051-8d83-27f982c1c9ea" path="/var/lib/kubelet/pods/c786f506-d163-4051-8d83-27f982c1c9ea/volumes"
Oct 31 13:17:11.750753 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:11.750668    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c54444cdb-4pldr_4814e92c-745a-480b-8b21-540d70d85cf6/maas-api/0.log"
Oct 31 13:17:12.293405 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:12.293347    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-8zvrv_8a257c14-3432-4628-b2e2-e2f3cf9f18c1/revoke-keys/0.log"
Oct 31 13:17:12.649777 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:12.649749    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e4f7bebc-7td2r_48a3d515-18f3-4084-8b25-60ca30514d21/revoke-keys/0.log"
Oct 31 13:17:12.986992 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:12.986914    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-wwt7t_8d3eae84-8c5d-4fd2-b983-1d3e34d1507b/postgres/0.log"
Oct 31 13:17:16.251396 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:16.251296    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cf7fb5f5-z927m_631d337f-1016-400a-99df-436efb0db2e0/kube-auth-proxy/0.log"
Oct 31 13:17:16.612925 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:16.612897    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-7q2ln_c8b74799-9fa1-41b6-99b1-6dc9a656841d/istio-proxy/0.log"
Oct 31 13:17:17.124254 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:17.124223    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-dswld_57ad591e-3b45-4312-8612-cf848dcf55e5/istio-proxy/0.log"
Oct 31 13:17:17.486002 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:17.485904    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-lsjpx_6cacf32b-0e2d-48cc-a0be-c99e26159bcc/istio-proxy/0.log"
Oct 31 13:17:22.474015 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:17:22.473977    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:17:28.024275 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:28.024234    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"]
Oct 31 13:17:28.027317 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:28.027288    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8zvrv"]
Oct 31 13:17:29.436971 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:29.436926    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a257c14-3432-4628-b2e2-e2f3cf9f18c1" path="/var/lib/kubelet/pods/8a257c14-3432-4628-b2e2-e2f3cf9f18c1/volumes"
Oct 31 13:17:49.023643 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:49.023553    5663 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"]
Oct 31 13:17:49.026589 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:49.026555    5663 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f7bebc-7td2r"]
Oct 31 13:17:49.436716 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:17:49.436683    5663 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a3d515-18f3-4084-8b25-60ca30514d21" path="/var/lib/kubelet/pods/48a3d515-18f3-4084-8b25-60ca30514d21/volumes"
Oct 31 13:17:52.474921 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:17:52.474886    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:18:22.475664 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:18:22.475625    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:18:40.632255 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632219    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q58mx/must-gather-d2wc4"]
Oct 31 13:18:40.632670 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632570    5663 cpu_manager.go:414] "containerMap: removing container" podUID="572e6bfe-c3af-4842-94fd-d5292b5d7092" containerName="payload-pre-processing"
Oct 31 13:18:40.632670 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632582    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572e6bfe-c3af-4842-94fd-d5292b5d7092" containerName="payload-pre-processing"
Oct 31 13:18:40.632670 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632596    5663 cpu_manager.go:414] "containerMap: removing container" podUID="ea478ebe-2d3d-4d38-a65d-d913915e532a" containerName="maas-api"
Oct 31 13:18:40.632670 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632601    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea478ebe-2d3d-4d38-a65d-d913915e532a" containerName="maas-api"
Oct 31 13:18:40.632670 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632610    5663 cpu_manager.go:414] "containerMap: removing container" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy"
Oct 31 13:18:40.632670 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632614    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy"
Oct 31 13:18:40.632670 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632635    5663 cpu_manager.go:414] "containerMap: removing container" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy"
Oct 31 13:18:40.632670 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632639    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy"
Oct 31 13:18:40.632670 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632645    5663 cpu_manager.go:414] "containerMap: removing container" podUID="32cdc561-19d5-4737-9a65-afc1fe919638" containerName="payload-processing"
Oct 31 13:18:40.632670 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632649    5663 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cdc561-19d5-4737-9a65-afc1fe919638" containerName="payload-processing"
Oct 31 13:18:40.632998 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632704    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy"
Oct 31 13:18:40.632998 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632711    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cdc561-19d5-4737-9a65-afc1fe919638" containerName="payload-processing"
Oct 31 13:18:40.632998 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632719    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="572e6bfe-c3af-4842-94fd-d5292b5d7092" containerName="payload-pre-processing"
Oct 31 13:18:40.632998 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632726    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea478ebe-2d3d-4d38-a65d-d913915e532a" containerName="maas-api"
Oct 31 13:18:40.632998 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.632827    5663 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ddde61-7a63-46a2-80a6-aaa3588e0c08" containerName="istio-proxy"
Oct 31 13:18:40.635692 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.635676    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q58mx/must-gather-d2wc4"
Oct 31 13:18:40.640007 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.639989    5663 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q58mx\"/\"default-dockercfg-7dg4w\""
Oct 31 13:18:40.640211 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.640190    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q58mx\"/\"openshift-service-ca.crt\""
Oct 31 13:18:40.640288 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.640275    5663 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q58mx\"/\"kube-root-ca.crt\""
Oct 31 13:18:40.659430 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.659406    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q58mx/must-gather-d2wc4"]
Oct 31 13:18:40.726333 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.726307    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwbgk\" (UniqueName: \"kubernetes.io/projected/305cb161-6198-452f-88b1-67c11e7cc8ab-kube-api-access-kwbgk\") pod \"must-gather-d2wc4\" (UID: \"305cb161-6198-452f-88b1-67c11e7cc8ab\") " pod="openshift-must-gather-q58mx/must-gather-d2wc4"
Oct 31 13:18:40.726472 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.726423    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/305cb161-6198-452f-88b1-67c11e7cc8ab-must-gather-output\") pod \"must-gather-d2wc4\" (UID: \"305cb161-6198-452f-88b1-67c11e7cc8ab\") " pod="openshift-must-gather-q58mx/must-gather-d2wc4"
Oct 31 13:18:40.827286 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.827250    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/305cb161-6198-452f-88b1-67c11e7cc8ab-must-gather-output\") pod \"must-gather-d2wc4\" (UID: \"305cb161-6198-452f-88b1-67c11e7cc8ab\") " pod="openshift-must-gather-q58mx/must-gather-d2wc4"
Oct 31 13:18:40.827459 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.827311    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwbgk\" (UniqueName: \"kubernetes.io/projected/305cb161-6198-452f-88b1-67c11e7cc8ab-kube-api-access-kwbgk\") pod \"must-gather-d2wc4\" (UID: \"305cb161-6198-452f-88b1-67c11e7cc8ab\") " pod="openshift-must-gather-q58mx/must-gather-d2wc4"
Oct 31 13:18:40.827605 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.827585    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/305cb161-6198-452f-88b1-67c11e7cc8ab-must-gather-output\") pod \"must-gather-d2wc4\" (UID: \"305cb161-6198-452f-88b1-67c11e7cc8ab\") " pod="openshift-must-gather-q58mx/must-gather-d2wc4"
Oct 31 13:18:40.839251 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.839224    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwbgk\" (UniqueName: \"kubernetes.io/projected/305cb161-6198-452f-88b1-67c11e7cc8ab-kube-api-access-kwbgk\") pod \"must-gather-d2wc4\" (UID: \"305cb161-6198-452f-88b1-67c11e7cc8ab\") " pod="openshift-must-gather-q58mx/must-gather-d2wc4"
Oct 31 13:18:40.969948 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:40.969875    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q58mx/must-gather-d2wc4"
Oct 31 13:18:41.245244 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:41.245164    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q58mx/must-gather-d2wc4"]
Oct 31 13:18:41.248896 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:18:41.248871    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod305cb161_6198_452f_88b1_67c11e7cc8ab.slice/crio-453aa7d84dae09cc78549357bdd8b429e4780ca6705b0343802c4b6a2aab2f38 WatchSource:0}: Error finding container 453aa7d84dae09cc78549357bdd8b429e4780ca6705b0343802c4b6a2aab2f38: Status 404 returned error can't find the container with id 453aa7d84dae09cc78549357bdd8b429e4780ca6705b0343802c4b6a2aab2f38
Oct 31 13:18:41.250880 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:41.250860    5663 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:18:41.851240 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:41.851204    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q58mx/must-gather-d2wc4" event={"ID":"305cb161-6198-452f-88b1-67c11e7cc8ab","Type":"ContainerStarted","Data":"453aa7d84dae09cc78549357bdd8b429e4780ca6705b0343802c4b6a2aab2f38"}
Oct 31 13:18:42.860022 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:42.859977    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q58mx/must-gather-d2wc4" event={"ID":"305cb161-6198-452f-88b1-67c11e7cc8ab","Type":"ContainerStarted","Data":"3a1586161801149418d9aed2bfb08be5354ef5c6a11fc797a2bf108af2792eb9"}
Oct 31 13:18:42.861384 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:42.861331    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q58mx/must-gather-d2wc4" event={"ID":"305cb161-6198-452f-88b1-67c11e7cc8ab","Type":"ContainerStarted","Data":"7bd8f89d802d5d9b84a4945c413c1397dc8892e6ff3720ba656e63982408ab5c"}
Oct 31 13:18:42.879490 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:42.879434    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-q58mx/must-gather-d2wc4" podStartSLOduration=1.9882068240000002 podStartE2EDuration="2.87941691s" totalImagesPullingTime="891.210086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:42.878347495 +0000 UTC m=+3047.618773712" watchObservedRunningTime="2026-10-31 13:18:42.87941691 +0000 UTC m=+3047.619843463"
Oct 31 13:18:44.084433 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:44.084399    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p8vf7_8c53a9b9-23b0-4598-8ee6-35de80dd12d4/global-pull-secret-syncer/0.log"
Oct 31 13:18:44.347831 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:44.347757    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-blk6p_0b8fc463-aafe-4750-aaeb-827522735a5e/konnectivity-agent/0.log"
Oct 31 13:18:44.403686 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:44.403654    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-137.us-east-2.compute.internal_0d0c123eab79dd4d273b2febcaaf1af3/haproxy/0.log"
Oct 31 13:18:52.501062 ip-10-0-140-137 kubenswrapper[5663]: E1031 13:18:52.500977    5663 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vcws" containerName="extractor"
Oct 31 13:18:53.479968 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:53.479939    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-647d9d5db7-nxnt2_1507e7d6-a6e4-4cb9-adf5-c24d8619cede/metrics-server/0.log"
Oct 31 13:18:53.556804 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:53.556766    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2rpgc_0589f9a7-c2b8-408f-8374-a25b489ba69a/node-exporter/0.log"
Oct 31 13:18:53.586400 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:53.586344    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2rpgc_0589f9a7-c2b8-408f-8374-a25b489ba69a/kube-rbac-proxy/0.log"
Oct 31 13:18:53.614990 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:53.614959    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2rpgc_0589f9a7-c2b8-408f-8374-a25b489ba69a/init-textfile/0.log"
Oct 31 13:18:54.036376 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.035829    5663 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"]
Oct 31 13:18:54.047335 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.047301    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.056397 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.055288    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"]
Oct 31 13:18:54.197140 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.197100    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ptf8z_a3ba234a-bb37-4ed4-a505-4a8494d4e862/prometheus-operator/0.log"
Oct 31 13:18:54.207974 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.207942    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3c57562-5af6-4d50-88a5-eb89e013fc25-podres\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.207974 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.207979    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3c57562-5af6-4d50-88a5-eb89e013fc25-sys\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.208252 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.208002    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3c57562-5af6-4d50-88a5-eb89e013fc25-lib-modules\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.208252 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.208135    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sltt\" (UniqueName: \"kubernetes.io/projected/f3c57562-5af6-4d50-88a5-eb89e013fc25-kube-api-access-4sltt\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.208252 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.208176    5663 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3c57562-5af6-4d50-88a5-eb89e013fc25-proc\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.218218 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.218193    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ptf8z_a3ba234a-bb37-4ed4-a505-4a8494d4e862/kube-rbac-proxy/0.log"
Oct 31 13:18:54.255903 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.255869    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-qzkwf_542d428b-b94d-4033-9805-d13da84ede85/prometheus-operator-admission-webhook/0.log"
Oct 31 13:18:54.286820 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.286748    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c46f6c557-tc9hn_be55aab1-5de3-442e-9f57-bc268d796440/telemeter-client/0.log"
Oct 31 13:18:54.308844 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.308812    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3c57562-5af6-4d50-88a5-eb89e013fc25-podres\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.309040 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.308868    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3c57562-5af6-4d50-88a5-eb89e013fc25-sys\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.309040 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.308903    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3c57562-5af6-4d50-88a5-eb89e013fc25-lib-modules\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.309040 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.308952    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3c57562-5af6-4d50-88a5-eb89e013fc25-podres\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.309040 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.308958    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sltt\" (UniqueName: \"kubernetes.io/projected/f3c57562-5af6-4d50-88a5-eb89e013fc25-kube-api-access-4sltt\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.309040 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.308971    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3c57562-5af6-4d50-88a5-eb89e013fc25-sys\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.309040 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.309024    5663 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3c57562-5af6-4d50-88a5-eb89e013fc25-proc\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.309040 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.309035    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3c57562-5af6-4d50-88a5-eb89e013fc25-lib-modules\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.309330 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.309121    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3c57562-5af6-4d50-88a5-eb89e013fc25-proc\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.319215 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.319187    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c46f6c557-tc9hn_be55aab1-5de3-442e-9f57-bc268d796440/reload/0.log"
Oct 31 13:18:54.321709 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.321673    5663 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sltt\" (UniqueName: \"kubernetes.io/projected/f3c57562-5af6-4d50-88a5-eb89e013fc25-kube-api-access-4sltt\") pod \"perf-node-gather-daemonset-vw7rf\" (UID: \"f3c57562-5af6-4d50-88a5-eb89e013fc25\") " pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.351081 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.351051    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c46f6c557-tc9hn_be55aab1-5de3-442e-9f57-bc268d796440/kube-rbac-proxy/0.log"
Oct 31 13:18:54.375437 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.375404    5663 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:54.674457 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.674410    5663 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"]
Oct 31 13:18:54.678813 ip-10-0-140-137 kubenswrapper[5663]: W1031 13:18:54.678772    5663 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf3c57562_5af6_4d50_88a5_eb89e013fc25.slice/crio-1d3eef3bd2a5755b6ab9dc7bd91563d471f81e7f2a10e856e83451102bcf87d7 WatchSource:0}: Error finding container 1d3eef3bd2a5755b6ab9dc7bd91563d471f81e7f2a10e856e83451102bcf87d7: Status 404 returned error can't find the container with id 1d3eef3bd2a5755b6ab9dc7bd91563d471f81e7f2a10e856e83451102bcf87d7
Oct 31 13:18:54.990244 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.990162    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf" event={"ID":"f3c57562-5af6-4d50-88a5-eb89e013fc25","Type":"ContainerStarted","Data":"5161999622aa9235e0de9871763af8cf1087b219aa8e66d7f84f0a1e3d60c3d5"}
Oct 31 13:18:54.990244 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.990230    5663 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf" event={"ID":"f3c57562-5af6-4d50-88a5-eb89e013fc25","Type":"ContainerStarted","Data":"1d3eef3bd2a5755b6ab9dc7bd91563d471f81e7f2a10e856e83451102bcf87d7"}
Oct 31 13:18:54.990450 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:54.990262    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:18:55.010716 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:55.010643    5663 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf" podStartSLOduration=1.010624066 podStartE2EDuration="1.010624066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:55.00862086 +0000 UTC m=+3059.749047077" watchObservedRunningTime="2026-10-31 13:18:55.010624066 +0000 UTC m=+3059.751050283"
Oct 31 13:18:56.743804 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:56.743775    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-555fc688cb-25l7p_f869ae4c-2b69-48e5-861b-425036ebe3b8/console/0.log"
Oct 31 13:18:56.787304 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:56.787273    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-j5n9s_5eb94f91-2be3-4a36-8d36-b3fdfb3bc026/download-server/0.log"
Oct 31 13:18:58.593390 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:58.593265    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pjdt2_f5bb3612-89f9-4494-9634-2b8bed367954/dns/0.log"
Oct 31 13:18:58.620028 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:58.619995    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pjdt2_f5bb3612-89f9-4494-9634-2b8bed367954/kube-rbac-proxy/0.log"
Oct 31 13:18:58.710411 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:58.710384    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bw8kq_ab9d44df-91e2-4e21-afd7-eac98bf106e5/dns-node-resolver/0.log"
Oct 31 13:18:59.407900 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:59.407850    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b6467d66c-vrqf6_be73966c-8b20-4130-8203-5cf6ca09df27/registry/0.log"
Oct 31 13:18:59.479142 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:59.479108    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9njc_fe072ecb-08a7-4c60-ad8b-d6643882ba73/node-ca/0.log"
Oct 31 13:18:59.969667 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:59.969634    5663 scope.go:122] "RemoveContainer" containerID="986785738c79e65a46fc95b45a2913bccb92e0485717067f9eee30ec2cdfbc2b"
Oct 31 13:18:59.981098 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:59.981065    5663 scope.go:122] "RemoveContainer" containerID="2c50f408782ac1c3099f345103f6f646ee8548c44c56006f1387bfad0052464d"
Oct 31 13:18:59.989800 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:18:59.989780    5663 scope.go:122] "RemoveContainer" containerID="11e87ed4d20b662370df5ed038b76e4a4ea04d249ee4882218f1064e178b5b35"
Oct 31 13:19:01.007323 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:01.007229    5663 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q58mx/perf-node-gather-daemonset-vw7rf"
Oct 31 13:19:01.363667 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:01.363618    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cf7fb5f5-z927m_631d337f-1016-400a-99df-436efb0db2e0/kube-auth-proxy/0.log"
Oct 31 13:19:01.439266 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:01.439230    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-7q2ln_c8b74799-9fa1-41b6-99b1-6dc9a656841d/istio-proxy/0.log"
Oct 31 13:19:01.535307 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:01.535258    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-dswld_57ad591e-3b45-4312-8612-cf848dcf55e5/istio-proxy/0.log"
Oct 31 13:19:01.608927 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:01.608899    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-lsjpx_6cacf32b-0e2d-48cc-a0be-c99e26159bcc/istio-proxy/0.log"
Oct 31 13:19:04.134578 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:04.134498    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7lv7c_609830fb-fa9a-4886-89be-cb40635b2c67/serve-healthcheck-canary/0.log"
Oct 31 13:19:04.807207 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:04.807171    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7vcws_adb879bf-aeb5-468c-8f93-4cb40d6e4e36/kube-rbac-proxy/0.log"
Oct 31 13:19:04.836581 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:04.836551    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7vcws_adb879bf-aeb5-468c-8f93-4cb40d6e4e36/exporter/0.log"
Oct 31 13:19:04.865618 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:04.865561    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7vcws_adb879bf-aeb5-468c-8f93-4cb40d6e4e36/extractor/0.log"
Oct 31 13:19:05.082848 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:05.082750    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cf7fb5f5-z927m_631d337f-1016-400a-99df-436efb0db2e0/kube-auth-proxy/0.log"
Oct 31 13:19:05.143129 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:05.143087    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-7q2ln_c8b74799-9fa1-41b6-99b1-6dc9a656841d/istio-proxy/0.log"
Oct 31 13:19:05.218528 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:05.218494    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-dswld_57ad591e-3b45-4312-8612-cf848dcf55e5/istio-proxy/0.log"
Oct 31 13:19:05.262814 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:05.262783    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-lsjpx_6cacf32b-0e2d-48cc-a0be-c99e26159bcc/istio-proxy/0.log"
Oct 31 13:19:07.690954 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:07.690916    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5967ccc7d4-dg7s4_e0f58e20-1eb3-4ad9-b689-2e6298f1ba57/manager/0.log"
Oct 31 13:19:07.714457 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:07.714427    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5967ccc7d4-dg7s4_e0f58e20-1eb3-4ad9-b689-2e6298f1ba57/copy-manifests/0.log"
Oct 31 13:19:09.028319 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:09.028289    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cf7fb5f5-z927m_631d337f-1016-400a-99df-436efb0db2e0/kube-auth-proxy/0.log"
Oct 31 13:19:09.068762 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:09.068735    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-7q2ln_c8b74799-9fa1-41b6-99b1-6dc9a656841d/istio-proxy/0.log"
Oct 31 13:19:09.147187 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:09.147150    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-dswld_57ad591e-3b45-4312-8612-cf848dcf55e5/istio-proxy/0.log"
Oct 31 13:19:09.185315 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:09.185285    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-lsjpx_6cacf32b-0e2d-48cc-a0be-c99e26159bcc/istio-proxy/0.log"
Oct 31 13:19:09.927951 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:09.927922    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dwk2j_dd96601d-214f-4e8f-9003-a78e49ba51a2/manager/0.log"
Oct 31 13:19:10.594558 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:10.594525    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-mt6xx_19431d95-2890-4c4c-b7b4-d7adb4fd51d5/manager/0.log"
Oct 31 13:19:10.906676 ip-10-0-140-137 kubenswrapper[5663]: I1031 13:19:10.906598    5663 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wd2j4_fbac8c23-7b52-45f5-b50f-649c72ac1fd7/manager/0.log"